Method for arranging workloads in a software defined automation system

ABSTRACT

Embodiments of a software defined automation (SDA) system that provides a reference architecture for designing, managing and maintaining a highly available, scalable and flexible automation system. A method is disclosed for arranging workloads in an SDA system including determining tasks of predetermined device functions, assessing industrial operational parameters for each task of the device functions; and ranking the tasks by the industrial operational parameters. The method continues by distributing tasks over automation devices based on the industrial operational parameters.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to and benefit from the followingprovisional patent applications: (1) U.S. Provisional Application Ser.No. 62/241,028 titled “Software-Defined Automation” filed on Oct. 13,2015, (2) U.S. Provisional Application Ser. No. 62/240,742 titled“Architecture for Connecting Objects in the Industrial Internet ofThings” filed on Oct. 13, 2015, (3) U.S. Provisional Application Ser.No. 62/348,770 titled “Software-Defined Automation” filed on Jun. 10,2016, (4) U.S. Provisional Application Ser. No. 62/354,683 titled“Software-Defined Automation Architecture” filed on Jun. 24, 2016, (5)U.S. Provisional Application Ser. No. 62/354,799 titled“Software-Defined Automation Architecture” filed on Jun. 26, 2016, and(6) U.S. Provisional Application Ser. No. 62/406,932 titled “SoftwareDefined Automation System and Architecture” filed on Oct. 11, 2016. Theentire contents of the aforementioned patent applications are expresslyincorporated by reference herein.

BACKGROUND

Automation is the use of automatic control devices and varioustechnologies to automate monitoring, operation and control of processesand installations without significant human intervention to achieveperformance that is superior to manual control. Known automation systemsfor monitoring and controlling processes and installations (e.g., inplants, buildings, etc.) typically comprise various automation devicessuch as controllers (e.g., Programmable Logic Controllers (PLCs),Programmable Automation Controllers (PACs)), input/output devices (I/Odevices), field devices (e.g., sensors and actuators), personalcomputers (PCs), Human Machine Interfaces (HMIs), and the like. Thecontrollers execute user-defined programs to control automatedprocesses. Typically, in a control system, controllers read input datafrom field devices such as sensors and metering devices and use theinput data to generate control outputs based on the user-definedprograms.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating aspects of a software definedautomation (“SDA”) technology in accordance with some embodiments.

FIG. 2A is a block diagram illustrating an example of a traditionalautomation system architecture implemented in some industries.

FIG. 2B is a block diagram illustrating an example of a simplified andflexible automation system architecture in accordance with someembodiments.

FIG. 3 is a block diagram illustrating an example of a flatter andflexible operational technology architecture for an enterprise inaccordance with some embodiments.

FIG. 4 is a diagram illustrating a simplified architecture of an SDAsystem in accordance with some embodiments.

FIG. 5 is a block diagram illustrating a functional architecture of SDAin accordance with some embodiments.

FIG. 6A is a block diagram illustrating subsystems of an SDA system inaccordance with some embodiments.

FIG. 6B is a diagram illustrating the scope of control of each of theSDA subsystems in accordance with some embodiments.

FIG. 7A is a block diagram illustration interaction between solutionsoftware and automation equipment in traditional automation systems andbetween a system software and automation equipment in an SDA environmentin accordance with some embodiments.

FIG. 7B is a block diagram illustrating example components of a systemsoftware of an SDA system in accordance with some embodiments.

FIGS. 7C-7F are screenshot diagrams illustrating example user interfacesof a system software in accordance with some embodiments.

FIG. 8A is a block diagram illustrating example fog server components inaccordance with a first embodiment.

FIG. 8B is a block diagram illustrating example fog server components inaccordance with a second embodiment.

FIG. 9A is a block diagram illustrating example components of a fogserver controller in accordance with some embodiments.

FIG. 9B is a block diagram illustrating example components of a computenode hosting virtual machines in accordance with some embodiments.

FIG. 9C is a block diagram illustrating example components of a computenode hosting containers in accordance with a first embodiment.

FIG. 9D is a block diagram illustrating example components of a computenode hosting containers in accordance with a second embodiment.

FIG. 9E is a block diagram illustrating example components of a computenode hosting a bare metal image.

FIG. 10A is a block diagram illustrating an example of a component viewof an SDA system in accordance with some embodiments.

FIG. 10B is a block diagram illustrating examples of a control view anda system view of an SDA system in accordance with some embodiments.

FIG. 11 is a block diagram illustrating an example of orchestration ofSDA subsystems to provision a functional unit on a compute node inaccordance with some embodiments.

FIG. 12 is a logic flow diagram illustrating an example method ofcreating an automation system in accordance with some embodiments.

FIG. 13A is a logic flow diagram illustrating an example method ofadding a functional unit to an automation system via a system softwarein accordance with some embodiments.

FIG. 13B depicts an example of a topological view of a conveyor systemin accordance with some embodiments.

FIG. 14 is a logic flow diagram illustrating an example method ofprovisioning a functional unit in an SDA system accordance with someembodiments.

FIG. 15 is a logic flow diagram illustrating an example method ofconfiguring a functional unit in an SDA system accordance with someembodiments.

FIG. 16A is a logic flow diagram illustrating an example method ofdefining an automation system via software accordance with someembodiments.

FIG. 16B is a logic flow diagram illustrating an example method ofcommissioning or provisioning a functional unit in an SDA system inaccordance with some embodiments.

FIG. 17A is a block diagram illustrating an example of an automationsystem in which an automation function can be configured in accordancewith a first embodiment.

FIG. 17B is a block diagram illustrating another example of anautomation system in which an automation function can be configured inaccordance with a second embodiment.

FIG. 17C is a block diagram illustrating yet another example of anautomation system in which an automation function can be configured inaccordance with a third embodiment.

FIG. 18 is a logic flow diagram illustrating an example of a method inaccordance with a first embodiment.

FIG. 19 is a logic flow diagram illustrating an example of a method inaccordance with a second embodiment.

FIG. 20 is a logic flow diagram illustrating an example of a part of themethods of FIGS. 18 and 19 in more detail.

FIG. 21 is a logic flow diagram illustrating an example of a part of themethods of FIGS. 18 and 19 in more detail.

FIG. 22 is a logic flow diagram illustrating an example of a part of themethods of FIGS. 18 and 19 in more detail.

FIG. 23 is a logic flow diagram illustrating an example of a part of themethods of FIGS. 18 and 19 in more detail.

FIG. 24 shows a diagrammatic representation of a machine in the exampleform of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

DETAILED DESCRIPTION 1. Overview

This disclosure describes Software-defined Automation (hereinafter“SDA”) technology and system (hereinafter “SDA system”) which provides areference architecture for designing, managing and maintaining a highlyavailable, scalable and flexible automation system.

This disclosure also describes methods for arranging workloads in an SDAsystem.

In some embodiments, the SDA technology enables control system(s) andassociated software to be run inside of a fog platform or a privatecloud. Control system(s) of varying degrees of complexity can be foundin traditional manufacturing facilities, refineries, submarines,vehicles, tunnels, baggage handling systems, energy management systems,building management systems, flood water control systems, grid controlsystems and the like. By moving the entire control system(s) or at leasta portion thereof to a fog platform or a private cloud, and providing asoftware interface to the control system elements, the SDA technologyenables engineering tasks over the whole lifecycle of automationengineering such as design, programming, configuration, installation,running, maintenance, evolution and shut down to be performed in asimpler, more efficient and cost effective way.

As depicted in FIG. 1, the architecture of an SDA system 100 comprisesof three aspects: (1) a smart distributed system 105, (2) acommunication backbone 110, and (3) smart connected devices 115. Thesmart distributed system 105 takes a software based approach to managevarious aspects of automation system(s) of an enterprise, throughout thelife cycle. This software based approach means the SDA system is easy tosetup, tune and adapt in regards to any evolving requirements to complyto changing business environment. In a smart distributed system,automation servers can host applications, databases, and the like andpermit a high level of elasticity. In some embodiments, the systemexhibits distributed intelligence by enabling a guest (e.g., acontrol/automation application) to be logically defined and distributedand re-distributed to run on one or more hosts (e.g., virtual machines,containers, bare metals) on a server, on a physical automationcontroller, an embedded system, and the like. The distribution can beinitiated for various reasons, e.g., to optimize performance, to upgradehardware, etc. For example, an application with heavy computationalrequirements can be deployed for execution on a compute resource that isable to provide the necessary computational resources. Similarly, anapplication with critical timing constraints can be deployed on acompute resource that is in close proximity to the field device itcontrols to reduce the impact of latency through network and/or otherdelays and improve the performance of the system.

The communication backbone 110 provides connectivity throughout theautomation architecture from the control level to fieldbus, in thecontroller's backplane, to the smart connected devices 115, and so on.This connectivity, enabled by Ethernet, greatly improves accessibilityof automation equipment and data and helps deliver the right informationto the right entity at the right time. In some embodiments, thecommunication backbone 110 of an SDA system can use one or morenetworking technologies such as Software Defined Networking (SDN), TimeSensitive Networking (TSN) and/or the like in some embodiments. SDNenables network elements which include switches and routers as well asany nodes taking on a similar role to be configured and re-configured ina simpler way without having to access each physical device. Forexample, each network device can be accessed by the logicallycentralized SDN controller using a set of protocols. TSN enablesreal-time Ethernet networking that enables real time control throughoutthe network.

Smart connected devices (or smart connected products) 115 are complexsystems that can connect to the network, generate data and execute afunction. Smart connected devices are aware of their operating context,and as such can make intelligent decisions and adapt their behavioraccordingly. For example, consider a sensor such as power meter that hasa basic function of sensing electrical networks. One or more functionsbesides the basic function can be deployed into the power meter totransform the power meter to a smart connected device. Such a smartconnected power meter can take advantage of its operating context to,for example, check for specific conditions, generate and send alarms,and the like. Smart connected devices 115 can comprise of hardware,software, sensors, storage, microprocessor(s), connectivity and thelike. Some non-limiting examples of smart connected devices include:controllers (e.g., programmable logic controllers or PLCs, programmableautomation controllers or PACs), drives, I/O concentrators, sensors,actuators, and the like.

An SDA system, in some embodiments, can be described as a collection ofservices. In some embodiments, it can be an infrastructure as a service(IaaS) providing virtual infrastructure on which customers can hosttheir own applications. It can also be a network as a service (NaaS) asit enables the network to be configured and re-configured or modified ina simple manner based on customer needs. The SDA system can also be asoftware as a service (SaaS) as it can host software (e.g., SoMachine,Unity) on one or more servers and enable a user to access the softwarein a client/server manner using a smart phone, laptop, personalcomputer, tablet, and/or other client device. It can also be adata/information as a service that defines data management atsolution/system level to avoid double definition and inconsistency andpermit big data and analytics. It can be a platform as a service (PaaS)providing a platform comprising a set of servers offering hosts toexecute on demand applications, embedded or not.

FIG. 2A depicts a traditional automation system architecture that iswidely implemented in many industries. In the traditional automationsystem architecture, automation devices at level 2 (e.g., PLCs 230A-C)are connected through device networks 235A-C to enable the automationdevices (e.g., field devices 225A-C) at level 1 to be controlled by thePLCs 230A-C respectively. Similarly, the PLCs 230A-C at level 2 andengineering stations 240 and process and historian servers 245 at level3 in the control room are connected to the same control network 250.This enables engineers to access and/or program PLCs 230A-C and accessprocess data stored in the historian servers 245 directly from theengineering stations 240. At level 4, at the top of the automationsystem architecture, the enterprise room can include system/enterpriseservers 260 which are connected to the engineering stations 240 andprocess and historian servers 245 at the control room level 210 throughthe enterprise network 255. Finally, at the highest level 5, the worldof industrial equipment, machines, controllers, sensors and actuators(“Operational Technology” or OT 265) spanning all four levels isintegrated with the office networks (i.e., Information Technology (IT)270).

The traditional automation system architecture (e.g., the traditional OTarchitecture 265 depicted in FIG. 2A) has several drawbacks. One suchdrawback is the locked down architecture. In other words, there is noflexibility in the traditional automation system architecture to makedynamic changes to configuration on the application, device or networkside. Moreover, the traditional automation system architecture ischaracterized by functional silos which create complexity and makecontrol systems inflexible. The complexity and lack of flexibility limitthe operational efficiency of the architecture, are a source offrustration to customers and require costly and inflexibleconfiguration. For example, in FIG. 2A, each of the functional units275A-C is depicted as having its own device network 235A-C respectively,which prevents different PLCs in different functional units frominteracting with each other. If there is a need to shift an applicationrunning in a PLC 230A in functional unit 275A to a PLC 230B infunctional unit 275B (e.g., because PLC 230A failed) and have thatapplication control the I/O device in function unit 275A, such a changewould require significant re-engineering and interruption of theindustrial operation, which can be costly.

Another problem with the traditional automation system architecture isthe complexity in terms of management of different applications anddevices as well as the network infrastructure. A typical automationsystem can comprise of hundreds of automation devices (or automationequipment) and processes managed by as many applications. For example,PLCs are programmed using a programming software (e.g., SchneiderElectric's Unity software for PLCs manufactured by Schneider Electric)and PLC configurations are stored in PLC software projects (e.g., Unityproject). Similarly, supervisory control and data acquisition (SCADA)configurations are stored in SCADA projects. Device configurations(e.g., IP addressing, I/O configuration, access control lists, localsub-components and supporting libraries, event triggering, passwords,and the like) are also generally managed through different softwareapplications. Similarly, automation device Internet Protocol (IP)configurations are not managed from a single point, but rather at eachpoint. Managing these applications and devices individually forcompatibility, versioning, maintenance, IP connectivity and so on isvery complex and requires a significant expertise and effort. Moreover,because these applications and devices are not centrally managed, thereis no way to recover the whole system in the event of disaster. As such,traditional automation system architectures are vulnerable to securityrisks (e.g., unauthorized changes to device configuration) and disasters(e.g., fire, flood).

Another downside to the lack of central management of applications anddevices is the difficulty in accessing data generated by different partsof the system. Aggregating large quantities of different types and setsof data generated by different applications and devices into one placebecomes a task that is too complex and time consuming. Without access torelevant data it becomes difficult to get a holistic view of the systemto make performance optimizations. For example, consider a scenariowhere a few devices on a plant floor may have resources available toexecute applications. Unless a plant engineer specifically accesses eachof those devices and makes a determination as to what resources areavailable, information about the resource availability from thosedevices will not be known and hence will not be considered when decidingwhere to deploy an application, or whether to add a new automationdevice. As a result, inefficient and sub-optimal decisions can be made.By way of another example, consider a situation where a virus infects anindustrial controller. In traditional automation systems, detection ofsuch an event can bring most of the plant, if not the entire plant, downbecause an engineer may have to physically swap out the controller witha new one and configure and program it again.

The SDA technology described herein overcomes these and other drawbacksof the traditional automation system architecture by transforming therigid, locked down traditional architecture into a flexible, “flatter,”software-defined architecture. The transformed OT architecture enablesnetwork configuration and automation function/application deployments onthe fly at the system level through the use of virtualization (e.g., ofservices, applications) technologies, configurable devices and/ornetworking technologies.

While the traditional automation architecture depicted in FIG. 2A isrigid and hierarchical with at least four levels of control, the exampleOT architecture as defined by the SDA technology depicted in FIG. 2B isconsiderably simpler, with three levels of control (hence the “flatter”description). These three levels of control include an enterprise roomlevel 205 (level 4), functional units, PLC, field process devices level280 (level 1) and a consolidated level 212 (level 3/4). The transformedOT architecture also comprises of an enterprise network 255 and a singledevice network 235 that replaces the fragmented device networks of thetraditional OT architecture. For example, as depicted in FIG. 2B, allthe automation devices such as the PLCs, 285, I/Os, HMIs 290A, 290B andengineering stations 240 are connected to a single device network 235.In this architecture, an application running in a PLC in functional unit275B can be moved to the server(s) 285 (e.g., by creating a virtual PLCwhich is a software implementation of a PLC on a host such as a virtualmachine or a container) and the network can be automatically configuredto ensure traffic from the virtual PLC (“vPLC”) in the server(s) 285flows to the I/O devices in functional unit 275B in a timely manner tomonitor and/or control input/output devices or field devices. Somenon-limiting examples of input devices include: sensors, meteringdevices, pressure switch, level switch, and the like. Similarly, somenon-limiting examples of output devices include: actuators, motors,relays or solenoids, analog devices, and the like. In this manner, SDAtechnology can simplify deployment and configuration of automationfunctions and/or applications.

One of the advantages of the disclosed SDA architecture is smartenterprise control. Smart enterprise control includes connectingexisting automation systems with other systems (e.g., enterprise,lifecycle and value chain systems) to optimize the entire manufacturingenterprise as a whole, and better enable the tangible benefits ofgreater business control. Smart enterprise control facilitates breakdown the silos of the enterprise and allows closer integration ofproduction systems with Enterprise Resource Planning (ERP) systems,Product Lifecycle Management (PLM) systems, Supply Chain Management(SCM) and Customer Relationship Management (CRM) systems. Thesedifferent enterprise systems have historically been managed somewhatindependently of each other, which prohibits a holistic view of theenterprise. The holistic approach of the disclosed SDA architecture canfacilitate an enormous efficiency gain for enterprises.

For example, smart connected devices can be tightly integrated with thewider enterprise to facilitate more flexible and efficientmanufacturing. Smart enterprise control is quite complex to implement,and the SDA architecture and standards enable the convergence ofinformation technology (IT) and operational transformation (OT) systems.Tighter integration allows enterprises to not only be more efficient,but also to have greater flexibility and responsiveness to volatilemarket conditions. The notion of control can expand from the real-timecontrol of a physical parameter, to the right-time control of the wholebusiness, including both physical and non-physical parameters. Examplebenefits to enterprises include the ability to increase protectionagainst cyber threats, be more innovative and be able to better managesafety, performance and environmental impact.

Some example applications of smart enterprise control includecustomization and lot sizes of one, reducing the size of productrecalls, detecting defect products earlier in the manufacturing processand modifying product design to eliminate root causes, modifyingproduction planning based on weather forecasts, modifying the productionplan/recipes based on the spot price of raw materials, and so on.

FIG. 3 is a block diagram illustrating an example of a flatter andflexible operational technology (“OT”) architecture for an enterprise inaccordance with some embodiments. As depicted the flatter OTarchitecture in accordance with the SDA technology has two layers: a“time-sensitive” IP-based cloud layer 330 for real-time deterministiccontrol and an enterprise cloud layer 325. The time sensitive layer 330encompasses sensors and actuators level 320 (L1) and discrete, hybrid orcontinuous control level 315 (L2) and is enabled by cloud computingtechnologies optimized for real-time deterministic communications. Inother words, the time sensitive layer 330 ensures that time sensitivecontrol/data traffic from L1 and L2 layers are managed to meet latencyand/or reliability requirements. As used herein, “cloud” refers to thetechnologies used, rather than the physical location of theinfrastructure. For example, in the automation industry, architectureswith one or more “on premise” clouds may be used.

In some embodiments, the OT devices that comprise the time sensitivelayer (e.g., sensors, actuators, and controllers in L1 and L2) arecloud-ready and capable of interfacing transparently with the IT systemsof the enterprise cloud layer. These devices can also have a high degreeof intelligence or logic in some embodiments. For example, controlvalves can be embedded with temperature, pressure and/or acousticsensors which are capable of operating autonomously using set pointsreceived from the enterprise cloud layer, for example, to determinetheir own needs for preventive maintenance, and/or inform themaintenance department in a timely manner.

The cloud enterprise layer 335 encompasses the manufacturing andoperations management (MOM) level 310 (L3) and the enterprise resourceplanning (ERP) level 305 (L4) of the hierarchy. ERP 335A, MOM 335B,product lifecycle management (PLM) 335C and other functions such asasset management 335D, energy management 335E, etc.) in the enterprisecloud layer 325 interoperate with each other and with the time-sensitiveindustrial automation and control systems to provide a coordinatedholistic view and management of the enterprise system. In someembodiments, information flow across both layers is completelytransparent using semantics and discovery mechanisms (e.g., based onIndustry standards).

The flatter architecture can provide many benefits for end users. Forexample, the flatter architecture is associated with a lowimplementation cost and increased flexibility. It can also supportconnectivity 340 to any end point, enabled by a standardized semanticinformation model. The semantic information model and associatedservices facilitate optimized push of field data to the cloud, andadapting of field device behavior based on analytics performed in thecloud.

Other benefits include implementation of additional incrementalfunctions, lot size 1, transparent and cost-effective connection toenterprise systems enabling information driven manufacturing.

Another benefit of the OT architecture in accordance with the SDAtechnology is its application to large scale control networkarchitectures. A large scale control network architecture is anengineering challenge for the whole lifecycle as it generally includes alarge number of devices connected over a network (e.g.,Ethernet/TCP-IP). The high number of connected devices means anunprecedented level of complexity. For example, such an architecture caninclude as many as 2800 PLCs and 5400 drives can be connected on 30network loops. The OT architecture in accordance with the SDA technologycan simplify the design, management and maintenance of such a largescale architecture. For example, in the OT architecture disclosedherein, data processing can be achieved in an organized and efficientmanner, which in turn optimizes operating performance. Response time,for example, with respect to data storage and retrieval, can bemonitored by an SDA system and adjustments can be made to optimize theoperating performance. Similarly, health of the components can bemonitored on a continuing basis by a centralized management componentand any events that could potentially impact the performance of thesystem can be detected in a timely manner and remedied via a coordinatedresponse on several fronts including virtualization, cyber security andnetwork. Similarly, the OT architecture can provide improved controlperformance by distributing processing and designing networksaccordingly taking into account various protocols for accessing deviceand application information. Moreover, system availability andsustainability can be improved by enabling fault diagnostics andredundancy.

These and various other aspects of the SDA system including variouscomponents, features, advantages and applications will now be discussedin detail.

2. SDA Architectures

A. Simplified Architecture

FIG. 4 is a diagram illustrating a simplified architecture of an SDAsystem in accordance with some embodiments. The architecture depicts afog server 405 linked to a system software 440, and smart connecteddevices 415A, 415B that are communicatively coupled to the fog server405 and the system software via a communication backbone 410. Thearchitecture also depicts that at least some smart connected devices415B and the fog server 405 can be communicatively coupled to a cloud450.

The fog server 405 is comprised of a collection of control resources andcompute resources that are interconnected to create a logicallycentralized yet potentially physically distributed system for hostingthe automation systems of an enterprise. The “fog server” or “fogplatform” as used herein is a cloud management system (or localizedsubsystem or localized system) that has been localized into one or morecompute and/or control nodes. In other words, the fog server 405 iscloud technology that has been brought down to the local ground orinstallation (hence the term “fog”) in the form of one or more computeand/or control nodes to manage the entire automation system or a portionthereof. The fog server 405 enables virtualization by providing avirtualization infrastructure on which automation system(s) can be runand/or managed. The virtualization infrastructure includes compute nodeswhich execute hosts such as virtual machines, containers and bare metals(or bare metal images). The hosts themselves can execute guests whichinclude applications and/or software implementations of physicalcomponents or functional units and an automation portal or systemsoftware 440. As used herein, virtualization is the creation of avirtual version of something. For example, a virtual component or avirtualized component (e.g., a virtual PLC, a virtual switch, networkfunction virtualization (NFV)) represents a function that is executed ona host running on a compute node. It does not have a physical existenceby itself. Fog server 405 need not be localized in a centralized controlroom; controllers, devices and/or servers 435 close to the sensors andactuators (e.g., IO device, embedded device) can also be consideredunder the management of the fog server 405. In some embodiments, the fogserver 405 can also aggregate, store and/or analyze data, and/or reportthe data or analytics to the cloud 450. The cloud 450 can be anenterprise cloud (i.e., private cloud), public or hybrid cloud. Thesystem software 440 provides a single entry point for an end user todefine (e.g., design, provision, configure, and the like) the automationsystem. One way to define the automation system is by managingdistribution of applications/application functions where users want themto be executed.

The smart connected devices 415A, 415B (also smart connected products)monitor and/or control devices, sensors and/or actuators close toequipment/raw materials/environment by executingapplications/application functions. In various embodiments, a smartconnected device has the following features: (1) physical and electricalcomponents, (2) firmware or a “smart” embedded part, and (3)connectivity and interoperability. In some embodiments, a smartconnected device can also have a cybersecurity component that may berunning remotely, or on board.

Some smart connected devices 415A can run applications or applicationfunctions (“applications”) locally (e.g., the speed/torque regulationloop of a speed drive) because they have the processing capability to doso. This means that there is no need to execute those applicationselsewhere (e.g., on a connected PC, a server or other computing devices)to get data to perform its functions. This has the advantage of fasterresponse time (i.e., less latency) and savings on network bandwidth.Another advantage of on-board or local execution of applications is thatit improves the consistency of data and the robustness of thearchitecture because the device can continue to produce information(e.g., alarm) or to log data even if the network is down.

In some embodiments, smart connected devices 415B can be wholly orpartially executed in one or more servers (e.g., server 435, fog server405). For example, a smart connected device 415B can be responsive toremote signals (e.g., remote method calls, application programminginterface or API calls) as if an application is running locally, when inactuality the application is running remotely, for example in the fogserver 405. In some embodiments, smart connected devices can capturereal-time data about its own state and the state of its environment(e.g., the devices it is monitoring) and send such data to the fogserver 405 and/or a remote cloud 450. In some embodiments, the smartconnected devices 415A, 415B can transform the captured real-time datainto information (e.g., alarms), store them and perform operationalanalytics on them. The smart connected devices 415A, 415B can thencombine both monitoring and controlling functions described above tooptimize own behavior and state.

The communication backbone 410 facilitates interaction between the fogserver 405, the system software 440 and the smart connected devices415A, 415B. The communication backbone (or the Internet of Things(IoT)/Industrial Internet of Things (IIoT) backbone) encompasses a setof network architectures and network bricks that enable physical andlogical connections of the smart connected devices 415A, 415B, the fogserver 405 and any other components that are part of the SDAarchitecture. For example, various equipment in a plant can be connectedto each other and with the enterprise system (e.g., IVIES or ERP) usingtechnologies based on various standards such as: Ethernet, TCP/IP, weband/or software technologies. The communication backbone in the form ofa unified global Industrial Ethernet backbone can provide: an easyaccess to the data, from the plant floor (OT) to the enterpriseapplications (IT), a flexible way to define different types of networkarchitectures (e.g., stars, daisy chain, ring) fitting with customerneeds, robust architecture that can fulfill requirements such asavailability, safety and harsh environment support and the rightinformation to the right people at the right time in one cable.

The communication backbone 410 includes a full Industrial Ethernetinfrastructure offering switches, routers and/or cable system to addressthe needs of all topologies. The communication backbone 410 alsosupports a set of connectivity protocols based on standards based onvarious standards (e.g., Modbus/TCP-IP, Ethernet IP, OPC UA, DHCP, FTP,SOAP, REST etc.). The communication backbone 410 can also support a setof web functions offering functions like diagnostic, monitoring andconfiguration using standard web pages and device integration referencearchitecture that defines the patterns, brick to integrate group ofdevices to controllers at application level and network level forconfiguration, tuning and diagnostic. In some embodiments, cybersecurity elements can be built in to the architecture. The communicationbackbone 410 also adheres to a set of architecture rules structuring thearchitecture at performances (Quality of Service or QoS), robustness(RSTP and PRP HSR for redundancy) and safety level (IEC61508). In someembodiments, the communication backbone 410 also supports integration ofa set of gateways to connect to legacy (i.e., non-Ethernet) equipment tothe network.

The communication backbone 410 can use multiple protocols to providemultiple services to fill multiple needs. Some examples of communicationneeds and suitable protocols are listed in table 1.

TABLE 1 Services and Protocols Service Protocol Device to deviceModbus/EtherNet/IP, DDS, OPC UA, pub/sub Device to control Modbus/Eip,NTP, DHCP, FTP Device to control for hard SercosIII, Profinet IRT,EtherCat real-time Control peer to peer DDS, OPC UA, pub/sub Control tocontrol room OPC, Modbus, TCP Across architecture Modbus/Eip, SNMP,SMTP, NTP, HTTP, FTP

The networks in the existing systems are very segmented to allowguaranteed or reliable communication. The communication backbone 410 inthe SDA architecture can overcome the problems of the existing systemsthrough Software Defined Networking (SDN) and Time Sensitive Networking(TSN) technologies. SDN technology enables separation of a network'scontrol logic from the underlying network hardware or device (e.g.,switches, routers) and logical centralization of network control. SDNtechnology can bring simplicity and flexibility in these networksallowing communication at and through different layers driven by networkpolicies. TSN technology adds a set of capabilities to standard Ethernetto provide real time capability and time guaranteed exchanges in areasor through the whole architecture. Moreover, cybersecurity solution canalso be integrated and adapted to the SDA architecture.

B. Functional Architecture

In some embodiments, the SDA architecture enables management of anautomation system via a set of controllers which provide system widemanagement of the resources. These controllers constitute the controlresources of the fog server and provide a homogenous method to managethe entire system. A system administrator can interact with thesecontroller nodes for the initial setup, system extension, diagnostic,maintenance, and the like. Similarly, applications executing in oroutside the system can interact with these controller nodes to managespecific facets or functions in the system (e.g., ICS tool, Networktool, electrical system tool), manage compute resources (e.g.monitoring, management of other applications and/or resources), and thelike. This functional view of the SDA architecture is depicted in FIG.5.

The example functional view of an SDA system depicted in FIG. 5 includesan application plane 575, a control plane 580 and a resource plane 582.The application plane 575 encompasses the system software 540 andsoftware components or applications 535 that are executed in the systemand which both use and manage a set of resources of the system. Thecontrol plane 580 includes a set of controllers including a fog servercontroller 510, an SDN controller 590A/TSN controller 590B (or networkcontroller) and a CS controller 555. These controllers provide astandardized set of interfaces to the applications in the applicationplane 575 to access and/or manage the resources in the resource plane582 of the system. In some embodiments, the controllers also providediagnostics, availability management, and the like. The SDN controller590A/TSN controller 590B manage and distribute network policies at thesystem level. Similarly, the CS controller 555 enforces securitypolicies 565 at the system level.

In some embodiments, these controllers can have a hierarchicalrelationship with one another. For example, an SDA system can include atop level controller (not shown) and a set of centralized controllers(e.g., fog server controller 510, the network controllers 590A, 590B andthe CS controller 555), each controlling a building or a site. The toplevel controller can for example distributes policies to the centralizedcontrollers to enable those controllers to control their own building orsite. The virtualization environment supports hierarchical distributionof the controllers.

The resource plane 582 can include network resources 560, computeresources represented by compute nodes 515, storage resources 525 andsecurity resources 595. The system software 540 and applications 535 areexecuted in computes nodes 515 managed by the fog server controller 510.The computes nodes 515 which provide the compute resources to the systemcan be physically distributed and managed by the fog server controller510. For example, some compute nodes in the form of servers are locatedin the fog server or private cloud while other compute nodes such assmart connected devices operate at the edge. Network resources 560 caneither be virtual network resources in the fog server, physicalinfrastructure resources in switching/routing hardware or infrastructureresources located in smart connected devices. Storage resources 525 canbe databases and/or other devices for storing virtual images, volumes,applications, process data, state data, and the like. Security resources595 can include security components residing on the compute nodes 515,storage nodes 525, and/or standalone components that provide securityservices such as enforcement of security policies, intrusion detectionand protection, and the like.

Controllers orchestrate and monitor some or all of the resources of thesystem. Applications managing the system (e.g., system software 540 orautomation portal, Network administration tool, etc.) send requests tothe system to apply specific strategies. For example, the systemsoftware can be used to deploy a new PLC connected to a set of deviceswith specific real time network requirements, security requirements andavailability/resilience requirements. In some embodiments, applicationscorrespond to software/firmware implementations of components. Theseapplications can be deployed on compute resources and can use storageresources and network resources to communicate.

3. SDA System

An SDA system comprises of various subsystems that work together toprovide a fully integrated solution for creating, managing and operatingautomation systems. FIG. 6A is a block diagram illustrating thesubsystems of an SDA system in accordance with some embodiments. An SDAsystem 600 in some embodiments includes a fog server subsystem 605 (“fogserver”) having a fog controller or redundant fog controllers 610, oneor more compute nodes 615 and storage 625. The SDA system 600 alsoincludes a software components subsystem 630. In other embodiments, theSDA system can further include a cybersecurity (“CS”) subsystem 650having a security controller or redundant security controllers 655,physical and/or virtualized security components 660 and a securitypolicy repository 665. In yet other embodiments, an SDA system can alsoinclude a network subsystem 670 having a network controller or redundantnetwork controllers 690, physical network 680, physical networkcomponents 682, virtual networks 620, virtual network components 622 anda network policies repository 685.

The fog server 605 provides a virtualization environment on whichautomation system(s) can be run and/or managed. The fog server 605comprises compute nodes 615 which provide logic processing capabilitiesand can host applications, databases and the like with a high level ofelasticity. Non limiting examples of compute nodes include: servers,personal computers, automation devices including smart connected devicesand the like.

The fog server controller 610 utilizes a fog server management softwareto perform its functions. The fog server management software can bebased on cloud management software such as OpenStack. Cloud managementsoftware such as OpenStack in their standard/off-the-shelf form aretypically used in the Information Technology (IT) world for data centermanagement. Automation system management, however, involves differentset of challenges. For example, some automation systems can runtime-critical and/or safety-critical applications that needdeterministic guarantees with respect to delay, reliability and/or otherfactors. Consider an automated cheese slicing system where a high speedsynchronized motion between a knife blade slicing through a cheese blockand the movement of the cheese block is critical to produce cheeseslices of uniform thickness. If there is any processing or networkdelay, it can result in cheese slices of different thickness, resultingin wastage and loss of productivity.

The fog server controller 610 manages all aspects of the virtualizationenvironment and the complete life cycle of the compute nodes 615. Forexample, the fog server 605 can stand up and stand down hosts such asvirtual machines, containers or bare metals on compute nodes, and createand destroy virtualized components 645 and virtual networks 620. Avirtualized component/element/instance 645, as used herein, is a logicalequivalent of a physical device or a portion of the physical device itrepresents, implemented as a software entity to run inside of the fogserver 605. Virtualized components 645 can also include softwarecomponents such as applications and/or application functions on a host(e.g., a virtual machine configured with an application is a virtualizedcomponent/element/instance).

The fog server controller 610 can provide high availability (HA) throughredundancy of the controller and management of compute node failures.The controller can also manage startup, shutdown and patching of theindividual compute nodes. In some embodiments, the fog server platformcan provide support for high availability of virtualized components. Insome embodiments, the fog server 605 can include a storage node or datastore 625. The storage 625 can store virtual images, volumes (i.e., harddrive of an instantiated image), application and process data, and thelike.

The software components subsystem 630 can include virtualized components645 that are hosted by the virtualization ecosystem of the fog server605. The software components subsystem 630 can also include virtualizedinstances of software 635 that run within the virtualization environment(e.g., software for programming, configuration, and/or management (e.g.,Unity, SoMachine, SCADA) that are used to program, configure, manage orotherwise interact with automation devices. In some embodiments, thesoftware component subsystem 630 can also include a system software 640(also called automation portal) that provides a single interface formanaging topology, inventory, configuration, programming, control,and/or diagnostics of the automation devices and/or the automationsystem as a whole.

Through the system software 640 users can access various applicationsfor system definition and system management over all life cycle phases.For example, the system software 640 can be used to configure andparametrize equipment during the engineering phase and tune, program,and/or diagnose equipment during the maintenance phase. Some of thebenefits of the system software 640 includes simplicity and ease for endusers and cost reduction as all aspects of any equipment in anautomation system can be managed from a single portal. In addition toproviding a single entry point to the entire system, the system software640 also presents a consistent user interface and user experience, whichhelp reduce inconsistency and increase efficiency and productivity.System software 640 and its components are described in detail inreference to system software 740 FIG. 7B.

The CS subsystem 650 includes an associated CS controller or redundantCS controllers 655 and virtualized and/or physical security components660. The security subsystem 650 provides a holistic cyber-securitysolution through security policies and security components such asintrusion detection/protection systems, virtualized next generationfirewalls, certificate authority and identification systems, and thelike. The CS controller 655 disseminates security policies to thevirtualized and/or physical components to ensure that necessary securityprotections are put in place. In some embodiments, the CS subsystem canalso provide security policy and authentication services to othercomponents and subsystems. The security policies of the CS system 650can be stored in a security policy repository 665 in some embodiments.

The network subsystem 670 includes the Ethernet network infrastructurefor the entire SDA system solution. In some embodiments, the networksubsystem 670 is an SDN network subsystem having an SDN controller orredundant SDN controllers as the network controller 690. The SDN networkprovides separation of network's control logic from the underlyingnetwork hardware (e.g., routers, switches) and logical centralization ofnetwork control through the SDN controller. This means that the SDNcontroller can disseminate network policies throughout the networkinfrastructure (i.e., physical network 680 and physical networkcomponents 682 as well as virtual networks 620 and virtual networkcomponents 622) to control connectivity, bandwidth and latency, ServiceLevel Agreements (SLAs) (e.g., re: deterministic response time/transfertime), traffic flow control, etc., and the network hardware canimplement those policies. The network policies of the network subsystem670 can be stored in a network policy repository 685 in someembodiments.

In some embodiments, the network subsystem 670 can comprise a mesh radionetwork. In mesh radio network, each node can connect to at least twoother nodes with data being passed from node to node in a process calledhopping. Because the nodes themselves serve as routers, radio meshnetworks typically do not require designated routers. However, some meshradio networks include one or more mesh routers along with the meshnodes to relay traffic on behalf of other mesh routers and/or meshnodes. In some embodiments, the network subsystem 670 can comprisevirtual circuits on a high speed radio frequency (RF) mesh or hybridnetwork with communication facilitated by only the radio transceivers ofthe nodes, without any external devices. Thus, in some embodiments,configuration of network elements of network subsystem or networkinfrastructure can include configuration of the mesh nodes and/or meshrouters (e.g., OpenFlow enabled mesh routers) in the mesh radio network.

In some embodiments, the network subsystem 670 can be a Time SensitiveNetwork (TSN) subsystem having a TSN controller as the networkcontroller 690 and TSN infrastructure. The TSN network subsystem ensuresthat mission critical and time-sensitive data are transferred/shared asper predefined maximum deterministic transfer time and with highreliability. Typically, TSN infrastructure includes TSN capable networkcomponents. It should be noted that in some embodiments, the networksubsystem 670 can comprise both SDN and TSN networks (and thus SDN andTSN controllers and SDN and TSN components). In various embodiments, thenetwork controller 690 can be a native fog server virtual networkcontroller, a traditional network management system controller, an SDNcontroller, a TSN controller, and/or any combination thereof.

The roles of the subsystems in the SDA solution complement each other toprovide a fully integrated solution. Specifically, the fog server 605can interact with each of these subsystems through hosting virtualizedelements of the subsystem and/or through the control functions of thesubsystem. While the fog server 605 has integral relationships with eachof the SDA subsystems, they are not considered within the scope of thefog server 605. FIG. 6B is a diagram illustrating the scope of controlof each of the SDA subsystems in accordance with some embodiments.

The realm of the fog server 605 is the fog server controller 610, thecompute nodes 615 and management of the virtualized components 645within the fog server 605. The virtualized components 645 and software635 (e.g., historian, SCADA, SoMachine, Unity) are not within the scopeof control of the fog server 605, but under the scope of control of thesoftware components subsystem 630. The software components 630, throughthe system software 640, however interact with the fog server controller610 and compute nodes 615 to provide configuration and control inputs tothe fog server 605 and/or other subsystems to drive their operation.

To provide a system wide solution, continuity of the network controlextends to include both the virtual and physical components of thenetwork. Therefore, the realm of the network subsystem 670 includes notonly the physical network components 682 and the physical network 680,but also the virtual networks 620 and the virtual network components 622which are created and exist within the fog server 605. This requiresfull integration between the network subsystem 670 and the fog server605 to provide the mechanisms to exercise this control. For example, thefog server controller 610 can create the virtual networks 620 in the fogserver 605 and control connectivity between the virtualmachines/containers hosted on the compute nodes 615 and the virtualnetworks 620, while the network controller 690 can configure the virtualnetwork components 622 of the virtual networks 620 in accordance withone or more network policies. This level of integration requiresorchestration of instantiation and deletion sequences as, clearly, thevirtual network 620 must exist before the virtual machines andcontainers can be connected.

The CS subsystem 650 has control over security components such asintrusion detection systems (IDS) 696A, intrusion protection systems(IPS) 696B (e.g., virtualized next generation firewalls) and the like aswell as the CS controller 655 that disseminates security policies todifferent entities. The CS subsystem 650 can be integrated with allaspects of the SDA system solution in some embodiments. For example, thenetwork controller 690 can utilize the security services provided by theCS subsystem 650 to provide security configuration information to thenetwork components (e.g., physical or virtual) within its scope. In someembodiments, the fog server 605 can utilize this service to authenticatelogins, provide security polices for host (virtual machine, container,bare metal) configurations, validate host images before instantiation,and the like.

In some embodiments, certain subsystems can be considered as beingexternal to the SDA system solution. These external subsystems includenon-SDN OT network and non-SDA edge devices 699 (e.g., legacy devices)and IT Network and back office equipment 698. In some embodiments,Industrial Internet of Things (IIoT) 697 or other a cloud based servicemay be considered external to or a part of the SDA system solution.

4. System Software or Automation Portal

FIG. 7A is a block diagram illustration interaction between solutionsoftware and automation equipment in traditional automation systems andin the SDA environment in accordance with some embodiments.

Typically, each type of equipment has its own specific software (alsocalled tool or software tool) using which the equipment can beconfigured, parameterized and/or programmed. For example, inmachine/manufacturing automation systems 706, solution software 735Asuch as SoMachine is used to configure, parameterize and/or programmachine equipment 701. Similarly, in process automation systems 708,another solution software 735B such as PlantStruxure PES (Process ExpertSystem) is used to configure, parameterize and/or program process. Atthe system level where automation equipment are more connected and moretightly integrated, it is highly inefficient for a user to manage thesesoftware solutions separately. In addition to the management concernssuch as keeping track of software solution versions, upgrading and soon, the separate software solution also means that it is not possiblefor a user to have a system view of all equipment, i.e., machineequipment and process equipment.

In an SDA system, a system software 740, through a common framework 742and other components, reconciles individual views in a system view. Inother words, the system software 740 provides a system level view of allautomation devices/equipment, taking into account the full automationscope. In the above example of an industrial automation system, thismeans that through the system software 740, a user can see the entiremachine 701 and process equipment 702, and can configure, parameterizeand/or program those machine and process equipment 701, 702 withouthaving to separately launch or invoke equipment type specific software.The common framework 742, in particular offers consistent userinterfaces, programming rules and infrastructure to simplifycommunication with the controllers (e.g., machine controllers 712, plantcontrollers 714), HMI 790, equipment 701, 702, and the like regardlessof whether they are machine or process related. In this manner, thesystem software 740 facilitates design, development and management of anautomation system as a whole.

FIG. 7B is a block diagram illustrating example components of a systemsoftware of an SDA system in accordance with some embodiments.

The system software 740 can be a web-based portal or a softwareapplication accessible from client devices. As used herein, clientdevices can include, but are not limited to: engineering stations,tablets 740A, mobile devices 740B, laptop computers 740C, desktopcomputers 740D, human machine interfaces (HMIs)/mobile HMIs 790, and thelike. As previously described, the system software provides a singleentry point through which a variety of SDA system managed automationdevices or equipment, whether they are in the fog server or on the plantfloor, can be configured, parameterized and programmed. Depending on theembodiments, the system software 740 can include more or lesscomponents. It should be noted that only select components of the systemsoftware 740 have been depicted for brevity.

The system software 740, in some embodiments, includes a commonframework 742 as described above. The common framework 742 can provideapplication interface(s) 752, controller/device interface(s) 754 anduser interface(s) 756 making tasks such as programming, configuration,tuning, diagnostics, etc., achievable from the system software userinterface, and more efficient.

In some embodiments, the system software 740 includes a topology viewgeneration component 726 which can collect topology information fromvarious parts of an automation system and render a system levelvisualization of all automation equipment, whether physical orvirtualized, and the links between them. In some embodiments, a topologyview of a portion of the automation system can be generated. Thetopology view can be a table view (e.g., shown in a navigation panel ofthe system software 740) or a chart view (e.g., shown in a design panelof the system software 740). The topology information can be collectedby querying components of the system software 740, the fog controller(e.g., fog server controller 410 in FIGS. 4A-4B, fog server controller610 in FIG. 6A), the network controller (e.g., the network controller690 in FIG. 6A, connections and existence of flows between components),and/or other subsystems of the SDA system in some embodiments.

The system software 740 can also include a functional unit templatelibrary 724 in some embodiments. The templates of functional units aresoftware models of functional units that can be parameterized andinstantiated in the fog server. A functional unit, as used herein, is ahardware entity, a software entity or a hybrid entity with hardware andsoftware portions capable of performing a specified purpose or function.It should be noted that a functional unit can be composed of otherfunctional units. For example, a PLC, a drive, a motor and an I/O modulecan each be considered a functional unit, and so can a conveyor beltsystem comprising of three PLCs, two IO modules, a drive and a motor.

In some embodiments, the system software 740 can include a set ofcomponents implementing domain specific logic or applications. Forexample, a parameterization component 728 can carry out parameterizationof equipment and functional unit templates described above (e.g., HMIparameterization). As used herein, parameterization includes setting ordefining properties. For example, a user can select an equipment from atopology view to parameterize. The parameterization component 728 canautomatically launch a parameterization interface (e.g., menu) of aparameterization software associated with the equipment. Likewise, aconfiguration component 732 can carry out configuration of equipment(e.g., motion drive configuration). As in the case of parameterization,a user can select an equipment from the topology view to configure. Inresponse, the configuration component 732 can display a configurationinterface of a configuration software associated with the selectedequipment. Similarly, a programming component 734 can launch theprogramming interface of a programming software associated with aselected equipment. A user can write or edit program code directly fromthe programming interface displayed in the system software withouthaving to launch the programming software. If the user wants to changethe program code of another equipment (e.g., an equipment of the sametype but different vendor, or completely different equipment type (e.g.,drive instead of PLC)) that uses a different programming software, theprogramming component 734 automatically identifies the equipment andlaunches the programming interface suitable for that equipment alongwith any program code associated with or currently deployed on theequipment. In some embodiments, the associations betweenequipment/equipment type and applications can be user-defined and storedin a storage node.

In some embodiments, the system software 740 can also include a set ofcomponents that support cybersecurity management, network management,data management, and/or other aspects of an automation system. Forexample, the network management component 716 can monitor automationequipment connected to the device and/or management networks (e.g., todiscover new devices as they connect to a network, to discover a devicethat goes offline). In some embodiments, the network managementcomponent 716 can also monitor network components such as switching androuting hardware that are part of the physical network.

The cyber security management component 718, in some embodiments, canmanage cyber security aspects of the automation system. For example, theCS management component 718 can create and maintain security profilesthat can be associated with any new functional unit or automationequipment in the automation system. The data management component 722 insome embodiments can manage how data is shared among the differentcomponents and equipment in the automation system. Typically, largequantities of different data are generated by different parts of thesystem. Pulling the large quantities of data into one place, andmanaging, organizing and displaying such data becomes a complex anddaunting task. The system software 740, through the data managementcomponent 722, solves this problem by aggregating data from thedifferent parts of the system in one place, making organization andanalysis of the data much more efficient. In some embodiments, the datamanagement component 722 can provide various filters which can beapplied to view select data associated with a specific equipment or asubset of equipment, without having to access different softwareassociated with different equipment. In some embodiments, the datamanagement component 722 can also manage and display in the systemsoftware environment, system variables which include data shared betweendifferent devices in the system and publishers of the data.

FIGS. 7C-7F are screenshot diagrams illustrating example user interfacesof the system software in accordance with some embodiments. FIG. 7Cdepicts an example screenshot of a user interface 750 of the systemsoftware 740 providing a graphical view of devices in an exampleautomation system. Through the system software, a user can manage theentire lifecycle of the system starting from design 752, configuration754 and programming 756. As depicted, the example automation systemincludes a PLC 758, a PLC 760 and a drive 240 among others.

In some embodiments, the system software enables different softwareapplications associated with the devices shown in the graphical view tobe directly accessed from the system software interface (or designview). For example, as depicted in the screenshot 751 of FIG. 7D, a usercan select the PLC 760 and click on “configure” from the menu 764. Thescreenshot 753 of FIG. 7E depicts a PLC configuration interface 768 ofthe PLC configuration application 766 that is launched in response tothe configure request. Similarly, an example configuration screen 770associated with the drive 762 depicted in FIG. 7C can be accesseddirectly from the system software as depicted in the screenshot 755 inFIG. 7F. In some embodiments, the code programmed into a device can alsobe accessed, edited and redeployed to the device directly from thesystem software.

5. Fog Server

FIG. 8A is a block diagram illustrating fog server components inaccordance with a first embodiment. The fog server is comprised of acontrol and management infrastructure called the controller nodes 810-1,810-2 along with the associated compute nodes 820-1, 820-2, 820-3, . . ., 820-N. Each of the compute nodes 820-1, 820-2, 820-3, . . . , 820-Ncan execute a number of hosts 802-1, . . . , 802-N and associatedvirtual networks 820. These hosts can be virtual machines, containers orbare metals. Each host in turn can execute a guest 804. A guest 804 caninclude an application, an application function (i.e., a piece orportion of an application corresponding to or performing a function), orany software implementation of a physical device, component orfunctional unit. In some embodiments, a host 802-1 can execute anotherhost 802-A which in turn can run a guest. For example, the host 802-1 ofcompute node 820-3 can be a virtual machine on which a container 802-Ais instantiated to run guest 804. The virtual networks 820 connect fromwithin the compute nodes (e.g., 820-1, 820-2, . . . ) through externalinterfaces (e.g., Ethernet ports) to the external physical networks(e.g., Data/OT network 865). Virtual networks 820 reside inside thecompute nodes (e.g., 820-1, 820-2, . . . ) and provide connectivitybetween the virtualized entities and the physical world. In someembodiments, a compute node can be a smart connected device, which canhave a physical part and a virtual part. For example, the compute node820-N can be a smart connected device 815 which can execute a host 802-Brunning a guest 804. The same smart connected device 815 can also have aphysical sensor/actuator 814. The compute node 820-N as other computenodes can connect to the data/OT network 865.

The guests 804 are not considered part of the fog server; however, themanagement of these entities is within the realm of the fog sever. Someof the management actions include distribution and re-distribution ofthe hosts, host instantiation, resource planning and management (e.g.,allocating RAM, network interfaces, and other resources), storageallocation, destruction, and the like.

While the virtual networks 820 are configured through services providedby the fog server, the responsibility for the orchestration of thesenetworks belongs to the network subsystem. This allows for cohesivenetwork management between the physical and virtual networks.

The fog server controller nodes 810-1, 810-2 are interconnected to thecompute nodes 820-1, 820-2, . . . , 820N via management network links812. These links may be physical with dedicated cabling or may belogical links on an underlying physical network. For example, link 812can be on the physical networks 806 or 865. By way of another example,links 806, 812 and 865 can share the same physical network, butdifferent logical networks. Use of technologies such as VLANs, VxLANS,VTNs, and the like, to provide logical separation of the physicalnetwork enables a single network to be used for multiple purposesconcurrently. In some embodiments, the fog server controller 810-2 canbe a redundant controller that provide high availability (HA)capability.

The storage node(s) 825-1/redundant storage node 825-2 can provide ahigh volume storage solution that is optimized for the type of accessand data and latency requirements needed to run an automation system.This node may be optional in some embodiments. The storage node(s) maybe incorporated into the system as storage node(s) directly connected tothe management network(s) 812 and/or OAM network(s) 806. If the storagenode is not provided, this role can be assumed by the controller nodes810-1, 810-2 and/or compute nodes 820-1, . . . , 820-N. The storagenodes can use redundancy to provide HA in some embodiments. It should benoted that in some embodiments, the storage node 825-1, 825-2 can be alogically centralized node comprising of other storage nodes that may bepotentially distributed.

FIG. 8B is a block diagram illustrating fog server components inaccordance with a second embodiment. This alternative deploymentscenario optimizes the hardware used to implement the fog sever. Thisdeployment scenario, known as Customer Premise Equipment (CPE) model,collapses the controller, storage, and compute functions into a singleserver device, i.e., CPE node 822-1. The CPE server node can also beduplicated (i.e., CPE node 822-2) to provide HA deployments in someembodiments. In this embodiment, the CPE server nodes may communicatethrough a management network 812. Storage node(s) 825 may beincorporated into the system as storage nodes(s) directly connected tothe management network(s) 812 and/or OAM network(s) 806 and/or datanetwork(s) 855. If the storage node is not provided, this role can beassumed by the CPE nodes 822-1 and 822-2. This scenario provides a lowcost solution that could be used in smaller deployment targets thataccept the constraint of not having distributed compute nodes.

FIG. 9A is a block diagram illustrating example components of a fogserver controller in some embodiments. As depicted, a fog servercontroller 910 can include a fog orchestration component 902 and a hostmanagement component 916 among others. The fog orchestration component902 interacts with the orchestration components of other subsystems ofan SDA system for provisioning, configuring, management, and the like.The role of the fog orchestration component 902 is discussed in detailin FIGS. 10B and 11.

In some embodiments, the host management component 916 can utilize oneor more host virtualization technologies to provide a virtualizationinfrastructure on which an automation system can be run and/or managed.For example, the host management component 916 can utilize hostvirtualization technologies to create virtualized instances of a device(e.g., software implementation of the device on a virtual machine),application or function in the automation system. The virtualized deviceruns as a software only instance in an environment that presents to thevirtual device an abstraction of the physical hardware isolated from thehost system. In addition to devices, various other aspects of theautomation system such as networks and security elements can also bevirtualized in some embodiments. Some of the host virtualizationtechnologies that can be utilized by the host management component 916are described in detail below.

A. Classic VM

FIG. 9B illustrates example components of a compute node hosting virtualmachines. In some embodiments, compute nodes 915 with virtualizationsupport can use virtual machines (VM) (host) 902-1, . . . , 902-N toprovide highly flexible, sandboxed applications 912 (guest). A computenode 915 hosts one or more virtual machines 902-1, . . . , 902-Nincluding the business logic of the application 912 and its ownOS/libraries 926. This mechanism provides a flexible application as theguest VM can be based upon any operating system 916 and can even useemulation to release constraints on hardware architecture. As such thevirtual machine can have its own virtual hardware. Indeed, because VMsare provided with direct access to the CPU through the hypervisor andeach classic VM has its own virtual hardware 924, kernel 922, initsystem 918, and OS 916, it is possible to run completely different OSes(e.g. Windows, Linux) on the same compute node concurrently, regardlessof the compute node's native OS. The penalty compared to the othersolutions (described below) may be in performance and determinism.Another downside may be the size of the application which could besubstantially larger as it must include a full kernel 922, init system918, an operating system 916, and associated libraries 914. Typicallyaccess to the physical hardware 932 is provided through a hypervisor 928which adds an additional layer and associated latency. Some vendorspecific accelerations can be used to mitigate this effect.

The virtual machines 902-1, . . . , 902-N can be migrated live, i.e.,running VMs can be migrated from one compute node to another with veryminimal impact to the running VMs and the associated applicationprocesses. This enables the host management component 916 and/or fogorchestration component 902 to provide a degree of load balancing, highavailability, and energy management by optimizing VM distribution amongmultiple compute nodes 915 and to shut down unneeded compute nodes.

B. Containers

FIGS. 9C and 9D illustrate example components of compute nodes hostingcontainers. Containers provide performance, flexibility and sizeimprovements for the applications but come with their own set ofconstraints. Containers use a memory sandbox that is supported by thehardware of the host machine to provide a secure and isolatedenvironment to run the application. Use of a container provides someperformance and size improvements over a VM as it directly uses thehost's drivers without the hypervisor layer. However, with containers,an application is inextricably linked to the host's hardwarearchitecture and kernel. One example application of containers is in ademand response scenario.

Referring to FIG. 9C, to achieve better performance, some containers904-1, . . . , 904-N can include only the application 912, while relyingupon the kernel 934, init system 918, operating system 916, andlibraries 914 native to the compute node. These containers have morelimitations from a libraries/application development point of view, butare lighter weight, smaller, faster to spawn, and are capable of thebetter performance.

Referring to FIG. 9D, some containers 907-1, . . . , 907-N can includethe full operating system 916 (minus the kernel) for the guestapplication 912, init system 918, and libraries 914 but run within thesandboxed container space of the host. Since the containers rely uponthe host's kernel 934 and its associated physical hardware 932, theymust also match the hardware architecture and kernel lineage of the host915.

Like VMs, containers can also be migrated live from one compute node toanother.

C. Bare Metal

FIG. 9D illustrates example components of a bare metal compute node. Insome embodiments, compute nodes 915 can serve as bare metal hosts toenable embedded systems to be managed by the fog server host managementcomponent 916. The bare metal hosts run a purpose built binary imagethat is tightly coupled to the host hardware 932—much like a traditionalembedded device. This binary image can take full advantage of the directaccess to the hardware 932 just as if the image were installed at thefactory. In some embodiments, similarly to how VMs are managed withinthe fog server, bare metal compute nodes can be provisioned andconfigured through the provisioning component 906 and configuringcomponent 908 of the host management system 916 in FIG. 9A.

In some embodiments, the bare metal image may be a complete kernel 934and OS 916 to turn the bare metal node into a full compute node with VMsand/or containers with its own support for VMs and/or containers.

Referring to FIG. 9A, the provisioning component 906 can create providerand/or tenant virtual networks and virtualized instances and connectthem together. The configuring component 908 can facilitateconfiguration of the virtualized instances and/or physical devices underthe management of the fog server. The data that is used forconfiguration can be received from the system software in someembodiments.

6. Orchestrations in an SDA System

FIG. 10A is a block diagram illustrating an example of a component viewof an SDA system in accordance with some embodiments. In the fog server(or the fog platform) 1005, one or more virtual devices 1036 andinstances of applications 1-N can run on one or more compute nodes (notshown) and/or edge devices depicted as a smart connected device 1015. Insome embodiments, analytics application(s) or engines 1006 can run on aremote cloud 1050 (e.g., cloud 450 in FIG. 4) as depicted, in the fogserver 1005 or both. In an industrial automation system, applicationsrelating to enterprise systems 1035 (e.g., Enterprise Resource Planning(ERP), Manufacturing Execution System (MES)) and asset management 1014can run at the enterprise room level (e.g., level 4, enterprise roomlevel 205 in FIG. 2B) or on the fog server 1005, while some on-premisesoftware 1008 (e.g., SCADA) can run in the fog server 1005. In abuilding automation system, the applications running at the enterpriselevel and at the fog server 1005 level can be of building managementsystems (not shown).

In some embodiments, a physical device 1034 may not have the capabilityto connect to the network to become a fog server managed device. Such adevice can still be managed and controlled via a cyber device 1032 thatis managed by the fog server 1005. This cyber device 1032 can be avirtual representation of one or more physical devices. The cyber device1032 can publish/subscribe to real time data in the fog server 1005 oralternatively can use point to point communication to get access to datafrom applications/devices managed by the fog server 1005. The cyberdevice 1032 can communicate with the physical device 1034 over an OTprotocol. The fog managed cyber device 1032 can thus be communicativelycoupled to a physical device 1034 via an OT protocol to form a softwaredefined machine 1046.

FIG. 10B is a block diagram illustrating examples of a control view andsystem view of an SDA system in accordance with some embodiments. TheSDA control view 1002 includes a system software 1040 and a number oforchestration components which ensure that each of the SDA subsystemswork in coordination with one another to define or commission and managethe automation system. The orchestration components include a fog serverorchestration component 1024, a network orchestration component 1022, acyber security orchestration component 1018 and a storage orchestrationcomponent 1016.

The SDA system view 1012, in some embodiments, includes a fog server1005 having a fog server controller 1010, one or more compute nodes 1015and storage 1025. In some embodiments, storage may be outside of the fogserver 1005 as depicted by storage 1026. The compute nodes 1015 andstorage 1025 in the fog server 1005 can be can be orchestrated togetherby the fog server orchestration component 1024 in some embodiments(i.e., fog server orchestration 1024 and storage orchestration 1026 canbe combined). While each one of the orchestration components areindividually orchestrated, a top level orchestration component—thesystem orchestration component 1016—orchestrates them together tovirtualize devices and applications on compute nodes 1015 in the fogserver 1005 (via fog server orchestration 1024), manage data associatedwith those virtualized devices and applications in storage 1025/1026(via storage orchestration 1026), define and disseminate cyber securitypolicies to all components of the SDA system (via cyber securityorchestration 1018), and network flows and communications (via networkorchestration 1022). A system software 1040 interfaces with the systemorchestration component 1016 to transform commands/instructions/signals(e.g., from user or another system) via fog server orchestration 1024,network orchestration 1022, cyber security orchestration 1018 and/orstorage orchestration 1026 into automation system changes. Moreover, thesystem software 1040 can run on the fog server 1005 and has a completeview of the automation system.

In some embodiments, network orchestration includes SDN orchestration(e.g., via SDN controller), TSN orchestration (e.g., via TSN controller)or SDN-TSN orchestration, which is a combination of SDN and TSNorchestrations (via both SDN and TSN controllers).

In some embodiments, the application instances running in the fog server1005 or on an edge device 1004 can share data using a communicationprotocol such as Data Distribution Service (DDS) or Open PlatformCommunications Unified Architecture (OPC-UA). DDS enables any equipmentconnected to the network 1042 to subscribe to any data produced by fogserver managed devices (e.g., device 1004, virtual devices/components inthe compute nodes 1015). The devices can update the subscribers in realtime by publishing the data value when those values change in someembodiments.

In other embodiments, data may be shared via point to pointcommunication. Regardless of the shared or point to point communicationprotocols used, data traffic to/from the application instances executingon virtual devices/components in the compute nodes 1015 are carried onvirtual networks 1020 which map to the physical network 1042. Similarly,data traffic to/from applications executing on physical devices arecarried by the physical network 1042.

FIG. 11 is a block diagram illustrating an example of orchestration ofSDA subsystems to provision a functional unit on a compute node inaccordance with some embodiments.

In some embodiments, a system software 1140 running an instance of anengineering toolchain enables a user to instantiate and manage an SDAsystem. An engineering tool chain can be specific to a targetedautomation system. For example, a toolchain targeted for an industrialautomation system would be different from one targeted for a buildingautomation system because those automation systems can have differenttypes of automation devices (and thus different device/functional unittemplates) as well as one or more software applications forparameterization, configuration, programming, and the like. Theengineering toolchain is integrated with a system (SDA) orchestrationcomponent 1116 via an application programming interface (API). So whenthe toolchain user issues a command, the toolchain drives the systemorchestration component 1116 in a way that causes the SDA system as awhole to work in coordination with one another to execute the command.

Consider a scenario in which the baggage handling capacity in an airportneeds to be increased by adding a new conveyer belt. A user can accessthe system software 1140 (loaded with a suitable toolchain) and select afunctional unit template, for example a template for a conveyer beltsystem, from a pick list and add it to the control system design panel.The user can parameterize the template to provide instance informationfor the new functional unit. For example, the conveyor belt template maycomprise of three virtual PACs, a number of IOs, a number of physicaland virtual switches. The user can provide instance information such asbut not limited to: instance identity (e.g., component/device names, IPaddresses, etc.), I/O connectivity (e.g., how the elements of thefunctional unit are connected, which I/O devices can the functional unitread from/write to), timing constraints (e.g., maximum deterministicresponse time or transfer time between the functional unit and anotherentity, e.g., the equipment it controls), security profiles (e.g.,ability to read/write access to data, ability to program the functionalunit), and the like. The functional unit description 1142, i.e., theinformation describing the functional unit template to be instantiatedis communicated by the system software 1140 to the SDA orchestrationcomponent 1116. In some embodiments, the functional unit description1142 can include information related to functional unit virtualizationdescription, communication flows, network flows, security profiles,and/or the like. By way of example, the functional unit virtualizationdescription can include the instance information including type andquantity of components to be instantiated or provisioned (e.g., 3 PLCs,2 distributed I/O modules, 1 virtual switch in the conveyer beltexample), redundancy requirements, and the like. The functional unitvirtualization description can also include, for each component,associated applications and version of applications, associatedprogramming package (e.g., Unity for the PLC) and the like to facilitateconfiguration and programming of the functional unit or the componentstherein.

The communication flow description can include information relating toI/O connectivity or links, I/O priority type (e.g., high priority, lowpriority), timing constraints, I/O list with connection information(e.g., data, rate), peer-to-peer data exchange, SCADA data exchange,other flows declarations (SNMP, Web, email, etc.), and the like.Security profiles can include access control lists (ACLs), port andprotocol lists, authorized bandwidth restrictions, black/white listedsites/addresses, and/or the like. In some embodiments, the functionalunit description 1142 can also include guest (e.g., virtual machine)configurations such as but not limited to: processor types, memory,affinity, virtual machine image validation and the like. Network flowdescription can include information such as bandwidth and port lists,flow path restrictions (e.g., no video or high bandwidth data on highpriority I/O links), port connectivity, interface speed, and the like.

The SDA Orchestration component 1116 parses the functional unitdescription into sub-descriptions and begins driving the orchestratorsof the various sub-systems accordingly. For example, the SDAorchestration component 1116 passes a description of the requestedcommunication flows 1144 extracted from the function unit description1142 to the cyber security orchestration component 1118 of the CScontroller 1155. The CS orchestration component 1118, based on therequested communication flows 1144, derives securities policies forhost/guest access, network traffic segmentation, firewallconfigurations, ACL configurations (e.g., IP address/name of connectionentity and nature of intended connection such as TCP/UDP port, allowedaccess types, block unauthorized protocols and ports, and the like),authorized logins for monitoring, configuration, and the like. Controlof traffic types allowed to an end points, configuring secure channels,controlling packet data length and addressing, and the like. In someembodiments, the various security policies can be managed by a securitypolices manager 1126. The authentication service 1128 in someembodiments can provide authentication service to the other subsystems.For example, it can authenticate requests to virtualize a functionalunit.

The cyber security orchestration component 1118, in some embodiments,provides necessary security policies for the fog server controller 1110and the network controller 1190 (e.g., SDN, TSN and/or other networkcontroller(s)) to the SDA orchestration component 1115. In otherembodiments, the CS orchestration component 1118 can cause thesecurities policies to be distributed directly to the relevantcontrollers. For example, security policies relating to virtualizationfunctions to the fog controller 1110, and security policies relating tothe network functions to the network controller 1190. In someembodiments, the CS controller 1155 can disseminate device and switchpolicy rules to the security protection system which can then managedeployment and enforcement of those policies at the device level.

The SDA orchestration component 1116 upon receiving the securitypolicies 1148 from the CS controller 1155, passes a description of thevirtualized elements of the functional unit extracted from thefunctional unit description 1142 and relevant security policies 1152 tothe fog orchestration component 1124. In some embodiments, the fogorchestration component 1124 can request the CS controller 1155 for therelevant security policies. The fog orchestration component 1124 drivesthe fog server controller 1110 (e.g., host management component 916 inFIG. 9A) to create, as necessary, the provider and/or tenant virtualnetworks 1120 in one or more compute nodes. This can includeinstantiating virtual switches or virtual routers. The fog orchestrationcomponent 1124 creates a virtualized instance of the functional unit1134 which includes creating a virtualized instance of each component inthe functional unit (i.e., 3 vPACs and 1 virtual switch in this example)and connecting the virtualized instances to the associated virtualnetworks 1120. In some embodiments, based on redundancy requirements(e.g., predefined or specified with the request), more than one instanceof the functional unit 1134 can be provisioned.

The SDA orchestration component 1116 passes a description of the networkflows 1154 associated with the functional unit and any required securitypolicies 1154 to the network orchestration component 1122. From thisdescription, the network orchestration component 1122 can discern therequired network paths, segmentation, and the like, and drive thenetwork controller 1190 to configure the network elements 1136 in thephysical network, as well as the network elements in the virtualnetworks 1120 accordingly. In some embodiments, all devices (e.g.,physical and virtual infrastructure and end devices) can request theirassociated security policies from a policy server 1138. In this manner,the SDA system can not only provision a functional unit on a computenode, but can also provision the network resources that the functionalunit needs in order to be in operation.

After the functional unit has been created or provisioned and thenetworking infrastructure is configured accordingly, the system softwarecan then be used to configure and program the components of thefunctional unit. For example, the vPACs of the functional unit can beconfigured and programmed using associated software through the systemsoftware portal to control the operation of the conveyor belt system. Insome embodiments, configuring the functional unit can also includeconfiguring the associated physical components of the functional unit.For example, the fog server controller 1110 can reconfigure an I/Omodule by updating its ACLs to allow the vPACs to connect. In someembodiments, the I/O module can be a smart connected device into whichthe fog server controller 1110 can program associated logic (e.g., logicfor processing security-based functionality).

7. Example Methodologies Implemented in the SDA System

FIG. 12 is a logic flow diagram illustrating an example method ofcreating an automation system in accordance with some embodiments.

At block 1202, a fog server subsystem that includes a fog servercontroller and multiple compute nodes creates or instantiates virtualcomponents of the automation system on one or more compute nodes (e.g.,via provisioning component 906 in FIG. 9A). The elements of theautomation system can be virtualized using virtualization technologiessuch as virtual machines, containers and bare metals. Moreover, thecompute nodes on which the virtual components are executed can bephysically distributed in some embodiments. For example, one computenode can be in the plant floor, while another compute node can be in acontrol room. Regardless of where the compute nodes are located, thecommunication between the fog server controller and the compute nodes isover a dedicated management network separate from the physical network,or over the same physical network.

At block 1204, the fog server subsystem (e.g., via provisioningcomponent 906 in FIG. 9A) creates associated virtual networks within thecompute nodes. At block 1206, the fog server subsystem (e.g., viaprovisioning component 906 in FIG. 9A) connects the virtual componentsto the virtual networks. The virtual networks are then connected to aphysical network. At block 1208, a network subsystem including a networkcontroller configures physical network components of the physicalnetwork and/or virtual network components of the virtual networks. Insome embodiments, the network subsystem configures the physical and/orvirtual network components by deploying network policies. The networkpolicies can include policies for controlling connectivity, bandwidth,latency and/or traffic flow. The network controller can be an SDNcontroller, a TSN controller or a combination thereof.

At block 1210, a CS subsystem that includes a security controllerdistributes security policies to the fog server subsystem and thenetwork subsystem for deployment to the virtual components executing onthe compute nodes and the physical and/or virtual network components. Atblock 1212, the fog server subsystem utilizes the physical and/orvirtual network components to communicate with physical components(e.g., field devices) of the automation system to control operation andmanagement of the automation system.

FIG. 13A is a logic flow diagram illustrating an example method ofadding a functional unit to an automation system via a system softwarein accordance with some embodiments.

Starting at block 1302, a user can launch the system software. At block1304, the system software can present a topological view of all thedevices, physical and virtual, that are managed by the automationsystem. FIG. 13B depicts an example of a topological view of a conveyorsystem that includes a PAC 1330 at the top of the hierarchy, a virtualPLC 1332 and associated I/O module 1334, a drive 1336, a motor 1338 anda conveyor (i.e., actuator) 1340. At block 1306, the system software canreceive a selection of a functional unit template (e.g., conveyor systemtemplate) to add to the automation system. The functional unit templatemay be selected from a template library in some embodiments. The systemsoftware can update the topological view to include the new functionalunit at block 1308. At block 1310, the system software can launch afirst application for configuring the functional unit. In someembodiments, configuring the functional unit can include informationsuch as but not limited to: IP addressing, I/O configuration, accesscontrol lists, local sub-components and supporting libraries, eventtriggering, passwords, and the like. At block 1312, the system softwarecan receive configuration data for the functional unit. At block 1314,the system software can launch a second application for system datamanagement. At block 1316, the system software can configure the newfunctional unit to receive/send data (e.g., via point to pointcommunication or through shared real time data bus). In someembodiments, configuration and data management can be performed throughthe same application. In such a situation, the system software canlaunch an application for configuration and data management of thefunctional unit at block 1318. The system software can receive theconfiguration data and/or instructions for data management at block1320. The system software can then configure the functional unit toreceive and/or send data at block 1322.

FIG. 14 is a logic flow diagram illustrating an example method ofprovisioning a functional unit in an SDA system accordance with someembodiments. At block 1402, the SDA system can receive a request tocreate or add a new functional unit to an automation system. In someembodiments, receiving the request can include receiving a selection ofa functional unit template from a library of functional unit templatesat block 1404. The selection can be made by a user via the systemsoftware user interface in some embodiments. In other embodiments, thedefinition of the new functional unit to be added to the automationsystem can be received from an entity that is communicatively coupled tothe system software (e.g., through an API). Receiving the request canalso include receiving information to parameterize the functional unittemplate at block 1406. At block 1410, the SDA system can authenticatethe request based on at least one security policy. In some embodiments,the authentication can be performed by the fog server subsystem usingthe at least one security policy from the cyber security subsystem. Atdecision block 1412, if the authentication is not successful, therequest can be denied by the SDA system at block 1416. Theauthentication step ensures that unauthorized changes to the automationsystem are not carried out by the SDA system.

If the request is successfully authenticated, the SDA system can createat least one virtual network in one or more compute nodes at block 1418,if a target virtual network does not exist. The SDA system can alsocreate a virtual instance of the functional unit at block 1420. Creatinga virtual instance of the functional unit includes creating a virtualinstance of each element of the functional unit. For example, if afunctional unit comprises of three PACs, virtualization of thefunctional unit would mean creation of three virtual PACs (vPACs). Atblock 1422, the SDA system can deploy the virtual instance of thefunctional unit on a compute node. At block 1424, the SDA system canconnect the virtual instance of the functional unit on the compute nodeto the virtual networks to provision or commission the functional uniton the compute node.

FIG. 15 is a logic flow diagram illustrating an example method ofconfiguring a functional unit in an SDA system accordance with someembodiments.

Once a functional unit has been created or provisioned (e.g., viaprovisioning component 906 in FIG. 9A), the functional unit can beconfigured using the system software. At block 1502, the SDA system(e.g., SDA system 600 in FIG. 6A) can receive configuration informationfor the new functional unit from the system software. At block 1504, theSDA system (via a network controller, e.g., network controller 690 inFIG. 6A) can determine at least one network path traversing throughvirtual and physical networks. The SDA system can configure one or morenetwork components in the at least one network path at block 1506.Configuring the network components can include providing and/orenforcing one or more network policies which specify how the networkcomponents are to direct different types of traffic flows. For example,a virtual/physical switch can be associated with a network policy thatspecifies allowing only HTTP traffic. So, the switch in operation wouldallow HTTP traffic to flow through, but other traffic such as MODBUStraffic would be blocked. At block 1508, the SDA system can configurethe virtual instance of the functional unit using the configuration data(e.g., via configuration component 908 in FIG. 9A). At block 1510, theSDA system can then allow data traffic to flow from the functional unitto a device (e.g., field device) via the at least one network path tocontrol an automated process.

FIG. 16A is a logic flow diagram illustrating an example method ofcommissioning or provisioning a functional unit in an SDA system inaccordance with some embodiments.

The example method includes creating, by a system controller (e.g., fogserver controller 910 in FIG. 9A) of a localized subsystem (e.g., fogserver subsystem), a virtualized instance of a functional unit of anautomation system in one or more compute nodes managed by the systemcontroller at block 1602. These compute nodes can include an automationsystem controller, a server, a personal computer, and/or a smartconnected device. In some embodiments, creating a virtualized instanceof a functional unit can include creating a fully virtualized instanceof the functional unit or partially virtualized instance of thefunctional. For example, if a functional unit includes two components(e.g., PLC 1 and PLC 2), then a fully virtualized instance of thisfunctional unit would include virtualization of both the components(i.e., two virtual components, for e.g., vPLC 1 and vPLC 2). Similarly,a partially virtualized instance of the functional unit could includevirtualization of one component (i.e., one virtual component, e.g., vPLC1), with the other component being a physical component (e.g., PLC 2).In some embodiments, the physical component can also be commissioned inthe SDA system (i.e., brought under the management of the fog server).The method of commissioning a functional unit having a physicalcomponent is described in reference to FIG. 16B.

The virtualized instance of the functional unit can be created from afunctional unit template selected from a library of functional unittemplates. A system software provides an interface for a user to accessthe library of functional unit templates to select the functional unittemplate and parameterize the functional unit template. Parameterizingthe functional unit template includes defining instance identity,input/output connectivity and security profile for the functional unitin some embodiments.

The system controller can create a virtual network in the one or morecompute nodes at block 1604, and then connect the virtualized instanceof the functional unit to the virtual network at block 1606. The virtualnetwork is mapped to a physical network to enable the virtualizedinstance of the functional unit to interact with a field device of theautomation system to control an automated process.

At block 1608, the system controller can configure security of thevirtualized instance of the functional unit by applying one or moresecurity policies from a cyber security subsystem. In some embodiments,this can include creating a virtualized instance of a securityprotection system (e.g., a virtual next generation firewall) on the oneor more compute nodes based on a security policy. In some embodiments,the virtualized instance of the functional unit includes one or morehosts on which software implementation of the functional unit isrunning. As such, configuring the security of the virtualized instanceof the functional unit can include configuring the security of: softwareimplementation of the functional unit, the one or more hosts, and/or theone or more compute nodes on which the one or more hosts are running. Insome embodiments, a host from the one or more hosts includes a virtualmachine, a container or a bare metal. In some embodiments, in responsiveto a request to create the virtualized instance of the functional unitof the automation system, the system controller can apply at least onesecurity policy to authenticate the request prior to creating thevirtualized instance of the functional unit. The security controller canalso apply at least one security policy to validate an image of eachhost associated with the virtualized instance of the functional unit.

At block 1610, the network controller of the network subsystem candetermine at least one network path from the virtualized instance of thefunctional unit to a field device through the virtual and physicalnetworks. Then at block 1612, the network controller can configure oneor more network elements in the at least one network path to enable flowof data traffic between the virtualized instance of the functional unitand the field device. At block 1614, the network controller canconfigure security of the one or more network elements in the at leastone network path by applying one or more security policies provided bythe cyber security subsystem.

FIG. 16B is a logic flow diagram illustrating an example method ofcommissioning or provisioning a functional unit in an SDA system inaccordance with some embodiments.

The example method includes receiving, by a system controller (e.g., fogserver controller 910 in FIG. 9A, fog server controller 610 in FIG. 6A),a commissioning request to commission a functional unit at block 1616.In response to the commissioning request, a network controller (e.g.,network controller 690 in FIG. 6A) responsive to receiving of thecommissioning request by the system controller, at least one networkpath for the functional unit that is connected to a physical network atblock 1618. At block 1620, the network controller configures one or morenetwork elements in the at least one network path to commission thefunctional unit in the automation system which enables data traffic flowbetween the functional unit and a field device in the automation system.

8. Workload Arrangements in the SDA System

The automation functions comprise of various functions and tasks thatneed to be arranged in a coordinated fashion in order to meet specifiedrequirements. The device functions are predetermined and assigned toautomation devices. As an example, a system for processing cement intopowder executes the high level function “create cement powder”. Thisfunction may be split up into device functions for a crusher, a conveyorbelt and a packaging machine. These functions may require supportingtasks as measuring temperature, speed, volume and productcomposition/mixture. Which measurements in turn are used for functionsof controlling throughput, process continuity, process stability andproduct quality. Some of these tasks may be critical for safety, somemay be time consuming and some may be resource consumptive.

In some embodiments, an SDA system can implement one or more methods forarranging workloads to optimize the performance of control systems(e.g., industrial control systems).

As an example of an automation system wherein an automation function isconfigured, FIG. 17A shows an engineering station 1701 for configuringwork stations, viz. a transport station controlled via a PLC 1702, anassembly station controlled via a PLC 1703 and a packing stationcontrolled via a PLC 1704. Next to the stations are local operatorscreens of the respective Human Machine Interfaces (HMIs) 1705-1707which are connected via Ethernet bus 1708 to top level HMI 1709. Theengineering system 1701 allows to design the automation function as adistributed and event driven application, which is deployed to the SCADAand Manufacturing Execution System 1710. Local engineering station 1711allows distribution of sessions for the PLCs of the work stations.Adapting IEC 61499, a standard for arranging communication betweenfunctional objects, advantage can be taken from this standard forsplitting the automation function into functional blocks and map anddistribute them over the PLCs of the work stations 1702-1704. As can beseen from elements T1-3, A1-5 and P1-4 which are downloaded from theengineering station 1701 to the respective work stations 1702-1704.

Another example of an automation system wherein an automation functionis configured is shown in FIG. 17B. This again shows engineering station1701 for configuring work stations. The automation system has atransport station controlled via a PLC 1702, an assembly stationcontrolled via a PLC 1703 and a virtual element 1712. The virtualelement 1712 may be implemented as a compute node, a smart device, aserver or other processing unit. In this example, the engineeringstation 1701 distributes the multiple instances of element T1, theelements A1-5 and the elements P1-4 in a different manner than in FIG.17A. The virtual element 1712 may be a standalone device providingsupport functions to the work stations controlled by PLCs 1702 and 1703.Or the virtual element 1712 may control a work station directly.

FIG. 17C shows yet another example of an automation system wherein allthe PLC-devices of FIG. 17A have been replaced by Virtual elements 1712,1713 and 1714 similar to the Virtual element 1712 of FIG. 17B. Thevirtual elements 1712, 1713, 1714 may be implemented as a compute node,a smart device, a server or other processing unit.

In order to allow a different distribution of these elements orfunctional processing tasks, hardware requirements thereof need to bematched to the capabilities of the available hardware. An applications'function (e.g. P1) characteristics, such as OS software environment,time sensitivity, network bandwidth, determine the requirements foroptimal guest parameters, as e.g. location. A host type parameter, e.g.a virtual machine, a container, or bare metal, determines what type ofconstraints are put on a guest that may be hosted. In turn, a computenode parameter is characterized by the type of hardware platform of thenode and determines what type of constraints are put on a host. The typeof hardware platform is determined by e.g. the processor type as ARM orX86; the memory size, the storage capacity, or the proximity.

9. Example Methodologies for Arranging Workloads

Referring to FIG. 18, a method is illustrated for arranging workload inthe SDA system (e.g., SDA system 600 in FIG. 6A). The SDA system that isprovided 1801 is configured via an automation portal or system software(e.g., system software 640 in FIG. 6A, system software 740 in FIG. 7B)for executing the intended automation function which includespredetermined device functions assigned to the various automationdevices.

As explained above, the automation function includes different devicefunctions for the various field devices. These device functions includevarious tasks that may be split according to the programmed softwareobjects. Accordingly, the method requires determining 1802 the tasks ofthe predetermined device functions. When the tasks have been determinedor identified, industrial operational parameters are assessed 1803 foreach task of the device functions. And when assessed, the tasks of thedevice functions may be ranked 1804 by the automation system operationalparameters. With the industrial operational parameters assessed andranked, tasks can be picked for distributing over the automation devicesbased on the industrial operational parameters. The distribution oftasks over the automation devices may include re-deploying tasks 1805 toat least one automation device and offloading tasks 1806 to one of thecompute nodes based on the industrial operational parameters. Inaddition, the automation device may be a smart connected device capableof hosting a compute node itself, thus allowing offloading of tasks tothe compute node hosted by the automation device.

The assessment 1803 of the industrial operational parameters provides anindication which tasks are relevant for which higher level function sucha safety, power consumption or other overall process performanceparameter. The ranking 1804 of the industrial operational parametersallows to evaluate which of the tasks may be re-deployed to fielddevices and which may be offloaded to compute nodes, while ensuring thatprocess performance requirements are met. The compute nodes may be partof a fog server.

The industrial operational parameters include several parameters thatmay be linked to overall process performance. A first industrialoperational parameter is a process critical level, which may express thelevel of availability and/or at least required, so the availability needof a function expressed as in e.g. uptime or extensivity of use. It mayalso express the level of required redundancy for a function, asredundancy is one manner in which an availability requirement may beachieved. It may further express a level of safety requirements, levelof guaranteed failure detections and/or fall back options. For example,an industrial steel processing plant may operate a furnace in a steelmill that under no circumstance may cool down. Furthermore, theindustrial operational parameter process critical level could also becompiled as a set of parameters, the set of parameters then includingthe level of redundancy, the availability need of the function, thesafety requirement and fall back options.

Another industrial operational parameter is a time sensitive level. Thismay express the level of required precision of execution time. It mayalso express a quantified time duration, e.g. a specified time frameduring which cement must remain in a mixer.

Furthermore, the industrial operational parameter time sensitive levelcould also be compiled as a set of parameters, the set of parametersthen including the precision of execution time and quantified timeduration.

A further industrial operational parameter is cost of execution. Thisindustrial operational parameter may express for example the processingtime constraints, such as the amount of processing time, it may expressthe level of resource consumption, and/or the processing capacity demandrequired for executing a specific task on a specific automation device.This may be expressed in the amount of memory required to storeinformation. It may also be expressed as amount of processing capabilityin e.g. millions of instructions per second (MIPS) or in floating pointoperations per second (FLOPS).

Furthermore, the industrial operational parameter cost of executioncould also be compiled as a set of parameters, the set of parametersthen including the processing time, the resource consumption, and theprocessing capacity demand.

Yet another industrial operational parameter is a proximity criticallevel. This parameter may express the required proximity of a task to aprimary actuator associated with the task and/or the proximityinterdependence of executed tasks, such as e.g. the interdependence of aprimary task regarding a secondary task.

For example, two tasks that interact frequently, or at least rely oneach other for intermediate results, may need to be computed atautomation devices or compute nodes in close proximity for the functionto be performed effectively. Close proximity may indicate in closedistance of each other in sense of physical space or of time. Yet it mayalso indicate that a third task requires proximity of two independenttasks, a chain of tasks. In particular, this may apply to proximity offield devices such as sensors and actuators. For example, forventilation control in a mineshaft, the task should be executed close tothe ventilator system. This in order to prevent operational interferenceor dysfunction in the occurrence of network problems, due to flooding orother natural disasters.

Furthermore, the industrial operational parameter proximity criticallevel could also be compiled as a set of parameters, the set ofparameters then including the interdependence of one task to multipleother tasks. So, the interdependence of the first task regarding asecond task, the interdependence of the first task regarding a thirdtask, the interdependence of the first task regarding a fourth task,etc.

One more possible industrial operational parameter to be applied is costperformance. This may express the capital expenditure as e.g. resourcecost. It may also express the operational expenditure, such as overallprocess reliability. For operational expenditures, an example is thecost of repair of a function in a hardly accessible area is higher, thanwhen close by in control room. For capital expenditure, an example isthe cost of running a function in high-end computer, which could behigher than when running on a small targeted platform.

Furthermore, the industrial operational parameter cost performance couldalso be compiled as a set of parameters. The set of parameters thenincluding the capital expenditure and operational expenditure.

Turning to FIG. 19, another example of a method for arranging workloadin an SDA system (e.g., SDA system of FIG. 4) is illustrated. Likewise,to the method described in relation to FIG. 19, the SDA system that isprovided 1901 may be configured through the system software (e.g.,system software 640 in FIG. 6A) for executing an automation function.Similarly, the method of this example includes determining 1902 thetasks of the predetermined device functions, assessing 1903 industrialoperational parameters for each task of the device functions, andranking 1904 the tasks of the device functions by the industrialoperational parameters. The SDA system may then be configured 1905according to the ranked industrial operational parameters, which mayinclude re-deploying tasks on automation devices and offloading tasks tocompute nodes.

The method then continues by re-assessing the industrial operationalparameters 1906 for each task and once more ranking 1907 theseparameters. When the industrial operational parameters are re-assessedand ranked, tasks may be re-deployed 1908 to automation devices andtasks may be off loaded to compute nodes of the automation system.

The second example described above shows the further advantage of themethod according to the embodiment, as it allows to re-arrange theworkload within the SDA system even after it has been configured toexecute one specific automation function. In this manner, when processcircumstances or process performance requirements are re-defined theworkload of the automation function may be arranged in a newconfiguration to meet the altered requirement specifications. Thus, themethod allows to more flexible manage and address performancerequirements in an industrial control system. As an example of a changein the automation function, a control function may be updated with ananalysis algorithm that would require more performance. Then thefunction could be split and some tasks could be moved to another computenode or device. Another example is expansion of hardware to expandcapacity, when the system was e.g. running at maximum capacity or beyondcapacity and the automation function needs re-balancing to preventadditional resources from remaining unused.

In the method, referring to FIG. 20, assessing the industrialoperational parameters for each task 2001 includes determining severalor all of the parameters for each task. Thus, the method includesdetermining at least one of the process critical level 2002, the timesensitive level 2003, the cost of execution 2004, the proximity criticallevel 2005; and the cost performance 2006. Not all parameters may needto be determined. But preferably for each task the same parameters aredetermined to allow comparison and ranking. Determining the parametersmay be done simultaneously or consecutively. Some of the parameters maybe set by an operator of the industrial control system, whereas othersmay be derived from the manner in which the tasks of the predetermineddevice functions are organized or programmed. Furthermore, theevaluation of parameters may include weighted factors, to allow acertain degree of flexibility in assessment of the parameters.

Similar considerations apply for the step of ranking tasks by industrialoperational parameters 2101. Accordingly, under reference to FIG. 21,the ranking of tasks includes ranking by process critical level 2102, bytime sensitive level 2103, by cost of execution 2104, by proximitycritical level 2105, and/or ranking by cost performance 2106.

Turning to FIG. 22, ranking the tasks by industrial operationalparameters 2201 may also include deselecting tasks as indication thatthese are not suited for re-distribution. That means that these taskswill not be evaluated and not considered for re-distribution.Accordingly, the ranking of tasks by parameters 2201 may includedeselecting process critical tasks 2202, deselecting time sensitivetasks 2203, ranking by cost of execution 2204, deselecting proximitycritical tasks 2205, and/or ranking by cost performance 2206. This mayin particular apply to the example described in relation to FIG. 19.

The manner in which the level of each industrial operational parameteris quantified may differ per parameter. One may be on a scale rangingfrom 1 to 100, from 0 to 10, or from 0 to 1 and include decimalfractions. For another it may be simply “yes/no” for re-deployment or itmay include a spectrum of colors. For yet another, it may be a setstrings of {“low”, “intermediate”, “high”}. In one example, theparameters may be set by an operator. In another example, they may beset by a system provider. In yet another example, they could becalculated based on processing power, time constraints of physicalconnections or other physical properties that may be determined in situ,even while the system is operational.

When the applicable parameters are assessed and the tasks are rankedaccording to these parameters, the re-deployment and offloading of taskscan be performed based on the industrial operational parameters. Thiswould allow a direct matching of task requirements and constraints tothe hardware capabilities of devices and compute nodes. For example, thesafety capability of a device must be higher than the safetyrequirement, so e.g. SIL3 when SIL2 is accepted. Or the availabilitycapability of one computing resource may be six-nines, which is acceptedwhen the requirement indicates five-nines. While matching tasks andhardware, a weighting factor may be applied for different parameters.For example, a timing precision specified as 20 msec may include apercentage of allowed deviation.

Referring to FIG. 23, the deploying of tasks to field devices andoffloading of tasks to compute nodes may include selecting tasks 2301for distribution based on rank and indicating selected tasks 2302 to thefog server controller node (e.g., fog server controller 610 in FIG. 6A).The fog server controller node will then distribute the selected tasks2303 to automation devices, such as PLCs, PACs, I/O devices or otherfield devices, based on the assessed industrial operational parameters.In turn, a network controller node (e.g., network controller 690 in FIG.6A) will set up network communication 2305 between the compute nodes tofacilitate execution of the distributed tasks. For example, byconfiguring a virtual network or controlling network elements of thephysical network.

In addition to the above, the method may further include selecting tasks2301 for offloading to compute nodes based on industrial operationalparameters. In which case, these tasks selected for offloading will alsobe indicated 2302 to the fog server controller node. The fog servercontroller node then distributes the tasks selected for offloading 2304to compute nodes based on the assessed industrial operationalparameters. And the network controller sets up network communication2305 between the compute nodes to facilitate execution of thedistributed tasks.

The distribution of selected tasks by the fog server controller node mayadvantageously also include the assessed industrial operationalparameters. As these may express a set of constraints or minimum ofrequirements that may be taken in account while matching the selectedtasks to available hardware capacities. Moreover, besides assigningcompute nodes for distributed tasks the fog server controller mayprovide i.e. set up new compute nodes.

10. Computer Systemization

FIG. 24 is a block diagram of an exemplary machine/computer/apparatusthat may perform various operations, and store various informationgenerated and/or used by such operations in accordance with someembodiments. The computer 2400 is intended to illustrate a hardwaredevice on which any of the entities, components or services depicted inthe examples of FIGS. 1-7B, 8A-11, 13B and 17A-C (and any othercomponents described in this specification) and methodologies describedin the examples of FIGS. 12-13A, 14-16B and 18-23 can be implemented,such as a server, client devices, compute nodes, controller nodes (e.g.,fog server controller (components 610, 810-x, 910, 1010, 1110), cybersecurity controller (e.g., components 655, 1155), network controller(e.g., components 690, 590A, 590B, 1190)), storage devices/nodes,databases, PLCs, PACs, and the like. The computer 2400 includes one ormore processors 2405 and memory 2410 coupled to an interconnect. Theinterconnect can represent any one or more separate physical buses,point to point connections, or both connected by appropriate bridges,adapters, or controllers.

The processor(s) 2405 is/are the central processing unit(s) (CPU(s)) ofthe computer and, thus, control the overall operation of the computer.In certain embodiments, the processor(s) accomplish this by executingsoftware or firmware stored in memory. The processor(s) may be, or mayinclude, one or more programmable general-purpose or special-purposemicroprocessors, digital signal processors (DSPs), programmablecontrollers, application specific integrated circuits (ASICs),programmable logic devices (PLDs), trusted platform modules (TPMs), orthe like, or a combination of such devices.

The memory 2410 is or includes the main memory of the computer. Thememory represents any form of random access memory (RAM), read-onlymemory (ROM), ternary content addressable memory (TCAM), flash memory,or the like, or a combination of such devices. In use, the memory maycontain a code. In one embodiment, the code includes a generalprogramming module configured to recognize the general-purpose programreceived via the computer bus interface, and prepare the general-purposeprogram for execution at the processor. In another embodiment, thegeneral programming module may be implemented using hardware circuitrysuch as ASICs, PLDs, or field-programmable gate arrays (FPGAs).

Also connected to the processor(s) through the interconnect are anetwork adapter 2425, a storage device(s) 2415 and I/O device(s) 2420.The network adapter provides the computer with the ability tocommunicate with remote devices, over a network and may be, for example,an Ethernet adapter or Fibre Channel adapter or wireless radio. Thenetwork adapter may also provide the computer with the ability tocommunicate with other computers within the cluster. In someembodiments, the computer may use more than one network adapter to dealwith the communications within and outside of the cluster separately.

The I/O device(s) can include, for example, a keyboard, a mouse or otherpointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. The display devicecan include, for example, a cathode ray tube (CRT), liquid crystaldisplay (LCD), or some other applicable known or convenient displaydevice.

The code stored in memory can be implemented as software and/or firmwareto program the processor(s) to carry out actions described above. Incertain embodiments, such software or firmware may be initially providedto the computer by downloading it from a remote system through thecomputer (e.g., via network adapter). In some embodiments, memory 2410and the storage device(s) 2415 can be a single entity.

The components introduced herein can be implemented by, for example,programmable circuitry (e.g., one or more microprocessors) programmedwith software and/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more ASICs, PLDs, FPGAs, etc.

Software or firmware for use in the SDA system introduced here may bestored on a machine-readable storage medium and may be executed by oneor more general-purpose or special-purpose programmable microprocessors.A “machine-readable storage medium”, as the term is used herein,includes any mechanism that can store information in a form accessibleby a machine.

A computer can also be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, a smart phone, atablet, a phablet, a processor, a telephone, a web appliance, a networkrouter, switch or bridge, a controller (e.g., PLC, PAC), or any machinecapable of executing a set of instructions (sequential or otherwise)that specify actions to be taken by that machine.

A machine-accessible storage medium or a storage device(s) includes, forexample, recordable/non-recordable media (e.g., ROM; RAM; magnetic diskstorage media; optical storage media; flash memory devices; etc.), etc.,or any combination thereof. The storage medium typically may benon-transitory or include a non-transitory device. In this context, anon-transitory storage medium may include a device that is tangible,meaning that the device has a concrete physical form, although thedevice may change its physical state. Thus, for example, non-transitoryrefers to a device remaining tangible despite this change in state.

The term “logic”, as used herein, can include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof.

11. Conclusion

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof, means any connection or coupling,either direct or indirect, between two or more elements; the coupling ofconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, shall referto this application as a whole and not to any particular portions ofthis application. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number respectively. The word “or,” in reference to alist of two or more items, covers all of the following interpretationsof the word: any of the items in the list, all of the items in the list,and any combination of the items in the list.

The above detailed description of embodiments of the disclosure is notintended to be exhaustive or to limit the teachings to the precise formdisclosed above. While specific embodiments of, and examples for, thedisclosure are described above for illustrative purposes, variousequivalent modifications are possible within the scope of thedisclosure, as those skilled in the relevant art will recognize. Forexample, while processes or blocks are presented in a given order,alternative embodiments may perform routines having steps, or employsystems having blocks in a different order, and some processes or blocksmay be deleted, moved, added, subdivided, combined, and/or modified toprovide alternative or sub-combinations. Each of these processes orblocks may be implemented in a variety of different ways. Also, whileprocesses or blocks are at times shown as being performed in series,these processes or blocks may instead be performed in parallel, or maybe performed at different times. Further any specific numbers notedherein are only examples: alternative implementations may employdiffering values or ranges.

The teachings of the disclosure provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various embodiments described above can be combined toprovide further embodiments.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the disclosure can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further embodiments of thedisclosure.

These and other changes can be made to the disclosure in light of theabove Detailed Description. While the above description describescertain embodiments of the disclosure, and describes the best modecontemplated, no matter how detailed the above appears in text, theteachings can be practiced in many ways. Details of the system may varyconsiderably in its implementation details, while still beingencompassed by the subject matter disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the disclosure should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the disclosure with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the disclosure to the specific embodimentsdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe disclosure encompasses not only the disclosed embodiments, but alsoall equivalent ways of practicing or implementing the disclosure underthe claims.

From the foregoing, it will be appreciated that specific embodiments ofthe disclosed system/technology have been described herein for purposesof illustration, but that various modifications may be made withoutdeviating from the spirit and scope of the embodiments. Accordingly, theembodiments are not limited except as by the appended claims.

1. A method for arranging workload in a Software Defined Automation(SDA) system, comprising: providing an SDA system, including a systemcontroller node and multiple compute nodes, wherein the multiple computenodes are communicatively coupled to the system controller node via afirst communication network; wherein the system controller node managesthe multiple compute nodes and virtualization of a control system or aportion thereof on at least one compute node from the multiple computenodes via the first communication network, the virtualized controlsystem including at least one virtualized control system elementconnected to a virtual network connected to a second communicationnetwork; wherein the at least one virtualized control system elementcontrols at least one physical control system element via the secondcommunication network connected to the virtual network; and wherein theSDA system is configured for executing an automation function comprisingpredetermined device functions assigned to the at least one automationdevice; determining tasks of the predetermined device functions;assessing industrial operational parameters for each task of the devicefunctions; ranking the tasks by the industrial operational parameters;distributing tasks over the at least two automation devices based on theindustrial operational parameters.
 2. Method according to claim 1,wherein the SDA system further comprises compute nodes; and whereindistributing tasks over the at least two automation devices comprises:re-deploying at least one task to the at least one automation devicebased on the industrial operational parameters; and/ offloading at leastone task to one of the compute nodes based on the industrial operationalparameters.
 3. Method according to claim 1, wherein the industrialoperational parameters comprise: a process critical level; and/or a timesensitive level; and/or a cost of execution; and/or a proximity criticallevel; and/or cost performance.
 4. Method according to claim 3, whereinassessing the industrial operational parameters comprises, for eachtask: determining the process critical level; and/or determining thetime sensitive level; and/or determining the cost of execution; and/ordetermining the proximity critical level; and/or determining costperformance.
 5. Method according to claim 4, wherein ranking the tasksby the industrial operational parameters comprises: ranking by processcritical level; and/or ranking by time sensitive level; and/or rankingby cost of execution; and/or ranking by proximity critical level; and/orranking by cost performance.
 6. Method according to claim 3, wherein theindustrial operational parameter process critical level comprises a setof parameters; comprising: availability need of function; and/or safetyrequirement.
 7. Method according to claim 3, wherein the industrialoperational parameter time sensitive level comprises a set ofparameters; comprising: precision of execution time, and quantified timeduration.
 8. Method according to claim 3, wherein the industrialoperational parameter cost of execution comprises a set of parameters;comprising: processing time constraints; resource consumption; and/orprocessing capacity demand.
 9. Method according to claim 3, wherein theindustrial operational parameter proximity critical level comprises aset of parameters; comprising: proximity to a primary actuator; and/orproximity interdependence of a primary task regarding a secondary tasks.10. Method according to claim 3, wherein the industrial operationalparameter cost performance comprises a set of parameters; comprising:capital expenditure; and operational expenditure.
 11. Method accordingto claim 2, wherein re-deploying tasks and offloading tasks comprises:selecting tasks for offloading based on industrial operationalparameters; indicating selected tasks for offloading to an SDAcontroller node; distributing the selected tasks for offloading tocompute nodes based on assessed industrial operational parameters;setting up network communication between the compute nodes to facilitateexecution of the distributed tasks.
 12. Method according to claim 11,wherein re-deploying tasks and off loading tasks further comprises:selecting tasks for re-deploying based on industrial operationalparameters; indicating selected tasks for re-deployment to the SDAcontroller node; distributing the selected tasks for re-deployment toautomation devices based on assessed industrial operational parameters;setting up network communication between the compute nodes to facilitateexecution of the distributed tasks.